2024-09-10T19:27:06.521 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T19:27:06.534 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:27:06.618 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899489 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} email: skanta@redhat.com first_in_suite: false job_id: '7899489' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN8c4jerr1332sLOuo6zAwc1wAix90FheaNyVQWDUTinC/zkorYaXsRrLXDohojTxMlbRR4MMYPLRryK6dskpk= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd-backfill teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T19:27:06.618 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T19:27:06.619 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T19:27:06.619 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T19:27:06.621 INFO:teuthology.task.internal:Checking packages... 2024-09-10T19:27:06.642 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T19:27:06.642 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T19:27:06.642 INFO:teuthology.packaging:ref: None 2024-09-10T19:27:06.642 INFO:teuthology.packaging:tag: None 2024-09-10T19:27:06.642 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:27:06.642 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:27:06.642 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:27:06.796 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T19:27:06.798 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T19:27:06.805 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T19:27:06.805 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T19:27:06.818 INFO:teuthology.task.internal:Saving configuration 2024-09-10T19:27:06.827 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T19:27:06.837 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T19:27:06.861 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899489', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 19:22:22.234065', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCN8c4jerr1332sLOuo6zAwc1wAix90FheaNyVQWDUTinC/zkorYaXsRrLXDohojTxMlbRR4MMYPLRryK6dskpk='} 2024-09-10T19:27:06.861 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T19:27:06.868 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-10T19:27:06.868 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T19:27:06.902 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f19f074fd00>, signals=[15]) 2024-09-10T19:27:06.902 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T19:27:06.909 INFO:teuthology.task.internal:Opening connections... 2024-09-10T19:27:06.909 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-09-10T19:27:06.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:27:06.986 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T19:27:06.993 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-09-10T19:27:07.067 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-09-10T19:27:07.067 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T19:27:07.125 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T19:27:07.126 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T19:27:07.126 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-09-10T19:27:07.145 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T19:27:07.153 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T19:27:07.160 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T19:27:07.161 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-09-10T19:27:07.181 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T19:27:07.188 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T19:27:07.188 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-09-10T19:27:07.337 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T19:27:07.365 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T19:27:07.379 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T19:27:07.379 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T19:27:07.379 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T19:27:07.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.397 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-09-10T19:27:07.454 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-503.el9.x86_64 2024-09-10T19:27:07.454 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.14.0-503.el9.x86_64 2024-09-10T19:27:07.455 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2024-09-10T19:27:10.033 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-10T19:27:11.775 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 34 MB/s | 20 MB 00:00 2024-09-10T19:27:18.192 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.5 MB 00:02 2024-09-10T19:27:20.501 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 17 kB/s | 19 kB 00:01 2024-09-10T19:27:21.514 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-10T19:27:26.878 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T19:27:28.980 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:27:28.995 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:27:28.995 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-10T19:27:28.995 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:27:28.995 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-09-10T19:27:28.996 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:27:28.997 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 79 M 2024-09-10T19:27:28.997 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 126 M 2024-09-10T19:27:28.997 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:27:29.204 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 245 kB/s | 23 kB 00:00 2024-09-10T19:27:29.597 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-09-10T19:27:29.988 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 36 MB/s | 29 MB 00:00 2024-09-10T19:27:30.297 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-10T19:27:30.297 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:27:30.297 INFO:teuthology.orchestra.run.smithi130.stdout:Total 61 MB/s | 79 MB 00:01 2024-09-10T19:27:30.979 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:27:31.336 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:27:31.336 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:27:32.271 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:27:32.271 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:27:34.734 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:27:35.435 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T19:27:35.523 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:27:38.310 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:27:38.444 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:27:43.261 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:27:43.303 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:27:48.154 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:28:26.709 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:28:26.758 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:28:27.182 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:28:27.182 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T19:28:27.182 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T19:28:27.182 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.024 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:28:28.025 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:28:28.380 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T19:28:28.467 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.468 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T19:28:28.468 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T19:28:28.468 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T19:28:28.468 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T19:28:28.468 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T19:28:28.468 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T19:28:28.468 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T19:28:28.527 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T19:28:28.527 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T19:28:28.527 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.06 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 82.el9 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2024-09-10T19:28:28.559 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Group : Unspecified 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 8274670 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : builder@centos.org 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:28:28.560 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T19:28:28.561 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T19:28:28.561 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T19:28:28.561 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2024-09-10T19:28:28.561 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:28:28.561 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T19:28:28.562 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T19:28:28.562 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 5.14.0-505.el9.x86_64 2024-09-10T19:28:28.562 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T19:28:29.172 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-09-10T19:28:30.476 INFO:teuthology.orchestra.run.smithi130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T19:28:30.508 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-09-10T19:28:30.510 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-09-10T19:28:30.526 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.PuuX9RkPuA 2024-09-10T19:28:30.526 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.PuuX9RkPuA 2024-09-10T19:28:30.594 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.PuuX9RkPuA 2024-09-10T19:28:30.732 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.PuuX9RkPuA is 6KB 2024-09-10T19:28:30.781 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.PuuX9RkPuA 2024-09-10T19:28:30.797 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T19:28:30.859 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T19:28:30.860 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T19:28:30.860 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T19:28:30.860 INFO:teuthology.orchestra.run.smithi130.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T19:28:30.861 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T19:28:31.036 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-09-10T19:29:01.040 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T19:29:01.040 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:29:01.041 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:29:19.421 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-09-10T19:29:28.429 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:29:28.430 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:30:28.490 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T19:30:40.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:30:40.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:30:40.864 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-09-10T19:30:41.218 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:30:41.218 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T19:30:41.218 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-09-10T19:30:41.234 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-505.el9.x86_64 2024-09-10T19:30:41.234 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T19:30:41.235 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T19:30:41.235 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T19:30:42.235 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T19:30:42.236 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T19:30:42.304 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-09-10T19:30:42.325 DEBUG:teuthology.parallel:result is None 2024-09-10T19:30:42.325 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T19:30:42.335 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T19:30:42.336 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T19:30:42.354 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T19:30:42.363 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T19:30:42.372 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T19:30:42.372 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T19:30:42.429 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T19:30:42.442 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T19:30:42.443 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T19:30:42.496 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:30:42.507 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:30:42.508 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T19:30:42.515 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T19:30:42.515 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T19:30:42.576 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T19:30:42.586 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T19:30:42.586 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T19:30:42.634 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T19:30:42.715 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T19:30:42.763 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T19:30:42.840 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T19:30:42.887 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:30:42.887 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T19:30:42.953 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-09-10T19:30:43.035 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:30:43.267 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T19:30:43.275 INFO:teuthology.task.internal:Starting timer... 2024-09-10T19:30:43.275 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T19:30:43.311 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T19:30:43.319 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-09-10T19:30:43.364 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-09-10T19:30:43.366 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T19:30:43.366 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-09-10T19:30:43.396 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-09-10T19:30:43.396 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi130.front.sepia.ceph.com': 'permissive'} 2024-09-10T19:30:43.396 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:30:43.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.462 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T19:30:43.462 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-09-10T19:30:43.529 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T19:30:43.538 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T19:30:43.555 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T19:30:43.557 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T19:34:24.411 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com')] 2024-09-10T19:34:24.413 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:34:24.414 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T19:34:24.492 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-09-10T19:34:24.575 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-09-10T19:34:24.576 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T19:34:24.590 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T19:34:24.591 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T19:34:24.591 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:34:24.645 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T19:34:24.659 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T19:34:24.683 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-09-10T19:34:24.693 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-09-10T19:34:24.708 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T19:34:24.721 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T19:34:24.777 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:34:24.780 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:34:24.781 INFO:teuthology.run_tasks:Running task install... 2024-09-10T19:34:24.789 DEBUG:teuthology.task.install:project ceph 2024-09-10T19:34:24.789 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T19:34:24.789 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T19:34:24.789 INFO:teuthology.task.install:Using flavor: default 2024-09-10T19:34:24.795 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T19:34:24.795 INFO:teuthology.task.install:extra packages: [] 2024-09-10T19:34:24.796 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T19:34:24.797 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:34:24.943 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T19:34:24.943 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T19:34:25.054 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T19:34:25.054 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:34:25.054 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T19:34:25.080 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T19:34:25.081 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T19:34:25.147 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T19:34:25.223 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T19:34:25.287 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-09-10T19:34:25.288 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-09-10T19:34:25.521 INFO:teuthology.orchestra.run.smithi130.stdout:45 files removed 2024-09-10T19:34:25.548 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw 2024-09-10T19:34:26.106 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 306 kB/s | 84 kB 00:00 2024-09-10T19:34:26.390 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-10T19:34:26.691 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 7.3 kB/s | 1.9 kB 00:00 2024-09-10T19:34:27.541 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.2 MB 00:00 2024-09-10T19:34:31.925 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 6.2 MB/s | 20 MB 00:03 2024-09-10T19:34:36.443 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 9.0 MB/s | 6.5 MB 00:00 2024-09-10T19:34:38.102 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-10T19:34:39.441 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 19 MB/s | 23 MB 00:01 2024-09-10T19:34:44.914 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-09-10T19:34:47.289 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:34:47.302 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:34:47.302 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:34:47.302 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T19:34:47.303 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T19:34:47.304 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:34:47.305 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:34:47.306 INFO:teuthology.orchestra.run.smithi130.stdout:Install 31 Packages 2024-09-10T19:34:47.306 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 2 Packages 2024-09-10T19:34:47.306 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:34:47.306 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 62 M 2024-09-10T19:34:47.306 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:34:48.257 INFO:teuthology.orchestra.run.smithi130.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-10T19:34:48.307 INFO:teuthology.orchestra.run.smithi130.stdout:(2/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 505 kB/s | 25 kB 00:00 2024-09-10T19:34:48.398 INFO:teuthology.orchestra.run.smithi130.stdout:(3/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 8.1 MB/s | 755 kB 00:00 2024-09-10T19:34:48.490 INFO:teuthology.orchestra.run.smithi130.stdout:(4/33): libradosstriper1-19.3.0-4858.gbad1f558. 5.4 MB/s | 509 kB 00:00 2024-09-10T19:34:48.607 INFO:teuthology.orchestra.run.smithi130.stdout:(5/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 13 MB/s | 11 MB 00:00 2024-09-10T19:34:48.658 INFO:teuthology.orchestra.run.smithi130.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 906 kB/s | 45 kB 00:00 2024-09-10T19:34:48.741 INFO:teuthology.orchestra.run.smithi130.stdout:(7/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_6 22 MB/s | 5.5 MB 00:00 2024-09-10T19:34:48.766 INFO:teuthology.orchestra.run.smithi130.stdout:(8/33): python3-ceph-common-19.3.0-4858.gbad1f5 1.3 MB/s | 143 kB 00:00 2024-09-10T19:34:48.800 INFO:teuthology.orchestra.run.smithi130.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 2.7 MB/s | 161 kB 00:00 2024-09-10T19:34:48.825 INFO:teuthology.orchestra.run.smithi130.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 5.4 MB/s | 321 kB 00:00 2024-09-10T19:34:48.850 INFO:teuthology.orchestra.run.smithi130.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 6.0 MB/s | 306 kB 00:00 2024-09-10T19:34:48.867 INFO:teuthology.orchestra.run.smithi130.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 2.3 MB/s | 99 kB 00:00 2024-09-10T19:34:49.076 INFO:teuthology.orchestra.run.smithi130.stdout:(13/33): ceph-common-19.3.0-4858.gbad1f558.el9. 17 MB/s | 22 MB 00:01 2024-09-10T19:34:49.318 INFO:teuthology.orchestra.run.smithi130.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 229 kB/s | 107 kB 00:00 2024-09-10T19:34:49.351 INFO:teuthology.orchestra.run.smithi130.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 352 kB/s | 171 kB 00:00 2024-09-10T19:34:49.377 INFO:teuthology.orchestra.run.smithi130.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 775 kB/s | 45 kB 00:00 2024-09-10T19:34:49.402 INFO:teuthology.orchestra.run.smithi130.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 492 kB/s | 160 kB 00:00 2024-09-10T19:34:49.461 INFO:teuthology.orchestra.run.smithi130.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 724 kB/s | 42 kB 00:00 2024-09-10T19:34:49.519 INFO:teuthology.orchestra.run.smithi130.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-09-10T19:34:49.553 INFO:teuthology.orchestra.run.smithi130.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 3.2 MB/s | 662 kB 00:00 2024-09-10T19:34:49.578 INFO:teuthology.orchestra.run.smithi130.stdout:(21/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-10T19:34:49.603 INFO:teuthology.orchestra.run.smithi130.stdout:(22/33): qatlib-service-24.02.0-1.el9.x86_64.rp 427 kB/s | 36 kB 00:00 2024-09-10T19:34:49.629 INFO:teuthology.orchestra.run.smithi130.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-10T19:34:49.739 INFO:teuthology.orchestra.run.smithi130.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.9 MB/s | 309 kB 00:00 2024-09-10T19:34:49.763 INFO:teuthology.orchestra.run.smithi130.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 217 kB/s | 26 kB 00:00 2024-09-10T19:34:49.788 INFO:teuthology.orchestra.run.smithi130.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-09-10T19:34:49.813 INFO:teuthology.orchestra.run.smithi130.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T19:34:49.847 INFO:teuthology.orchestra.run.smithi130.stdout:(28/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 14 MB/s | 839 kB 00:00 2024-09-10T19:34:49.872 INFO:teuthology.orchestra.run.smithi130.stdout:(29/33): re2-20211101-20.el9.x86_64.rpm 3.2 MB/s | 191 kB 00:00 2024-09-10T19:34:49.981 INFO:teuthology.orchestra.run.smithi130.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-09-10T19:34:50.073 INFO:teuthology.orchestra.run.smithi130.stdout:(31/33): librados2-19.3.0-4858.gbad1f558.el9.x8 17 MB/s | 3.5 MB 00:00 2024-09-10T19:34:50.131 INFO:teuthology.orchestra.run.smithi130.stdout:(32/33): libarrow-9.0.0-11.el9.x86_64.rpm 8.4 MB/s | 4.4 MB 00:00 2024-09-10T19:34:50.256 INFO:teuthology.orchestra.run.smithi130.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-10T19:34:50.261 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:34:50.261 INFO:teuthology.orchestra.run.smithi130.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-10T19:34:50.374 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:34:50.410 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:34:50.410 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:34:51.104 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:34:51.104 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:34:52.594 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:34:52.676 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T19:34:52.711 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T19:34:52.736 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T19:34:52.759 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T19:34:52.781 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T19:34:52.969 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T19:34:53.023 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T19:34:53.388 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T19:34:53.471 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:34:53.546 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:34:53.637 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T19:34:53.664 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T19:34:53.735 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T19:34:53.772 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T19:34:53.815 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T19:34:53.885 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T19:34:53.907 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T19:34:53.971 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T19:34:54.006 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T19:34:54.117 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T19:34:54.148 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T19:34:54.190 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T19:34:54.221 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T19:34:54.250 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T19:34:54.311 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T19:34:54.352 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T19:34:54.680 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T19:34:54.722 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T19:34:54.813 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T19:34:54.853 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T19:34:54.897 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T19:34:55.383 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T19:34:55.408 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T19:34:55.463 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T19:34:55.540 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T19:34:55.568 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T19:34:58.140 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:34:58.181 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:34:58.749 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T19:34:58.783 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T19:34:58.879 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T19:34:58.879 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T19:34:58.879 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T19:34:58.879 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:34:58.918 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:35:09.262 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-09-10T19:35:09.263 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:10.281 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T19:35:10.306 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T19:35:10.307 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:10.307 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T19:35:10.307 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:35:10.307 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:35:10.307 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:10.308 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T19:35:10.392 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T19:35:10.392 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:35:11.568 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:35:11.568 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T19:35:11.568 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T19:35:11.568 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T19:35:11.568 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T19:35:11.569 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T19:35:11.570 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T19:35:11.571 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T19:35:11.571 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T19:35:11.571 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T19:35:12.480 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T19:35:12.480 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:12.480 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-09-10T19:35:12.481 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.481 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.481 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T19:35:12.485 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.486 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:12.487 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:12.825 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-test 2024-09-10T19:35:13.388 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:29 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:13.524 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T19:35:13.525 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:13.526 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 50 M 2024-09-10T19:35:13.527 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 215 M 2024-09-10T19:35:13.527 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:14.004 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 736 kB/s | 161 kB 00:00 2024-09-10T19:35:14.630 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 103 kB/s | 64 kB 00:00 2024-09-10T19:35:14.654 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 350 kB/s | 304 kB 00:00 2024-09-10T19:35:15.288 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 33 MB/s | 50 MB 00:01 2024-09-10T19:35:15.290 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:15.290 INFO:teuthology.orchestra.run.smithi130.stdout:Total 29 MB/s | 50 MB 00:01 2024-09-10T19:35:15.318 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:15.338 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:15.338 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:15.718 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:15.718 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:16.254 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:16.323 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T19:35:16.368 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T19:35:16.398 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T19:35:21.139 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T19:35:21.220 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T19:35:23.671 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T19:35:23.671 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T19:35:23.671 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T19:35:23.671 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T19:35:24.005 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T19:35:24.005 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:24.005 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:24.005 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:24.006 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:24.006 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T19:35:24.006 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T19:35:24.006 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:24.006 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:24.273 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph 2024-09-10T19:35:24.867 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:40 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:25.014 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:25.016 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:25.016 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:35:25.016 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:25.016 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T19:35:25.017 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T19:35:25.018 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T19:35:25.019 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout:Install 37 Packages 2024-09-10T19:35:25.020 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:25.021 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 31 M 2024-09-10T19:35:25.021 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 109 M 2024-09-10T19:35:25.021 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:25.537 INFO:teuthology.orchestra.run.smithi130.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 48 kB/s | 6.6 kB 00:00 2024-09-10T19:35:25.696 INFO:teuthology.orchestra.run.smithi130.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 5.2 MB/s | 1.5 MB 00:00 2024-09-10T19:35:25.796 INFO:teuthology.orchestra.run.smithi130.stdout:(3/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 6.0 MB/s | 2.4 MB 00:00 2024-09-10T19:35:25.847 INFO:teuthology.orchestra.run.smithi130.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 5.7 MB/s | 291 kB 00:00 2024-09-10T19:35:26.014 INFO:teuthology.orchestra.run.smithi130.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.5 MB/s | 253 kB 00:00 2024-09-10T19:35:26.130 INFO:teuthology.orchestra.run.smithi130.stdout:(6/37): python3-cryptography-36.0.1-4.el9.x86_6 11 MB/s | 1.2 MB 00:00 2024-09-10T19:35:26.289 INFO:teuthology.orchestra.run.smithi130.stdout:(7/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-10T19:35:26.315 INFO:teuthology.orchestra.run.smithi130.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 580 kB/s | 106 kB 00:00 2024-09-10T19:35:26.373 INFO:teuthology.orchestra.run.smithi130.stdout:(9/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 5.7 MB/s | 4.8 MB 00:00 2024-09-10T19:35:26.398 INFO:teuthology.orchestra.run.smithi130.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.5 MB/s | 125 kB 00:00 2024-09-10T19:35:26.449 INFO:teuthology.orchestra.run.smithi130.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T19:35:26.474 INFO:teuthology.orchestra.run.smithi130.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 734 kB/s | 135 kB 00:00 2024-09-10T19:35:26.499 INFO:teuthology.orchestra.run.smithi130.stdout:(13/37): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-09-10T19:35:26.524 INFO:teuthology.orchestra.run.smithi130.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 692 kB/s | 35 kB 00:00 2024-09-10T19:35:26.549 INFO:teuthology.orchestra.run.smithi130.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-10T19:35:26.575 INFO:teuthology.orchestra.run.smithi130.stdout:(16/37): python3-urllib3-1.26.5-6.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-09-10T19:35:26.600 INFO:teuthology.orchestra.run.smithi130.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 676 kB/s | 51 kB 00:00 2024-09-10T19:35:26.625 INFO:teuthology.orchestra.run.smithi130.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 553 kB/s | 42 kB 00:00 2024-09-10T19:35:26.684 INFO:teuthology.orchestra.run.smithi130.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-09-10T19:35:26.709 INFO:teuthology.orchestra.run.smithi130.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-09-10T19:35:26.742 INFO:teuthology.orchestra.run.smithi130.stdout:(21/37): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-09-10T19:35:26.768 INFO:teuthology.orchestra.run.smithi130.stdout:(22/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-09-10T19:35:26.802 INFO:teuthology.orchestra.run.smithi130.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 676 kB/s | 18 kB 00:00 2024-09-10T19:35:26.827 INFO:teuthology.orchestra.run.smithi130.stdout:(24/37): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-09-10T19:35:26.852 INFO:teuthology.orchestra.run.smithi130.stdout:(25/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-09-10T19:35:26.877 INFO:teuthology.orchestra.run.smithi130.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 314 kB/s | 23 kB 00:00 2024-09-10T19:35:26.903 INFO:teuthology.orchestra.run.smithi130.stdout:(27/37): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-09-10T19:35:26.920 INFO:teuthology.orchestra.run.smithi130.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 292 kB/s | 20 kB 00:00 2024-09-10T19:35:26.936 INFO:teuthology.orchestra.run.smithi130.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 784 kB/s | 46 kB 00:00 2024-09-10T19:35:26.962 INFO:teuthology.orchestra.run.smithi130.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-10T19:35:26.979 INFO:teuthology.orchestra.run.smithi130.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 391 kB/s | 16 kB 00:00 2024-09-10T19:35:27.004 INFO:teuthology.orchestra.run.smithi130.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-09-10T19:35:27.029 INFO:teuthology.orchestra.run.smithi130.stdout:(33/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-09-10T19:35:27.046 INFO:teuthology.orchestra.run.smithi130.stdout:(34/37): python3-tempora-5.0.0-2.el9.noarch.rpm 536 kB/s | 36 kB 00:00 2024-09-10T19:35:27.063 INFO:teuthology.orchestra.run.smithi130.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-10T19:35:27.096 INFO:teuthology.orchestra.run.smithi130.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-10T19:35:27.130 INFO:teuthology.orchestra.run.smithi130.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-09-10T19:35:27.134 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:27.134 INFO:teuthology.orchestra.run.smithi130.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-10T19:35:27.284 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:27.324 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:27.324 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:27.792 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:27.792 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:29.150 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:29.208 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T19:35:29.249 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T19:35:29.277 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T19:35:29.349 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T19:35:29.392 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T19:35:29.425 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T19:35:29.517 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T19:35:29.587 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T19:35:29.629 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T19:35:29.663 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T19:35:29.696 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T19:35:29.730 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T19:35:29.763 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T19:35:29.803 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T19:35:29.847 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T19:35:29.881 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T19:35:29.921 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T19:35:29.953 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T19:35:30.036 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T19:35:30.172 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T19:35:30.253 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T19:35:30.306 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T19:35:30.344 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T19:35:30.393 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T19:35:30.457 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T19:35:30.641 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T19:35:30.690 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T19:35:30.768 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T19:35:30.856 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T19:35:30.902 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T19:35:31.033 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T19:35:31.196 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T19:35:31.224 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:35:31.251 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:32.715 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T19:35:32.742 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T19:35:32.742 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:32.742 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T19:35:32.742 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:35:32.743 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:35:32.743 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:33.199 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:35:33.223 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:33.449 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T19:35:33.473 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T19:35:33.473 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:33.473 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T19:35:33.473 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:35:33.474 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:35:33.474 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:33.538 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T19:35:35.150 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T19:35:35.151 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T19:35:35.152 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T19:35:35.153 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T19:35:35.153 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T19:35:36.209 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T19:35:36.209 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:36.209 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:36.209 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:36.209 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T19:35:36.210 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T19:35:36.211 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:36.212 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:36.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base 2024-09-10T19:35:37.153 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:37.243 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:35:37.286 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:37.287 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:35:37.287 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:37.351 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install cephadm 2024-09-10T19:35:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:38.048 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 784 k 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 790 k 2024-09-10T19:35:38.049 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:38.305 INFO:teuthology.orchestra.run.smithi130.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-10T19:35:38.306 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:38.306 INFO:teuthology.orchestra.run.smithi130.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-10T19:35:38.307 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:38.311 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:38.311 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:38.317 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:38.318 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:38.389 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:38.767 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:35:38.867 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:35:39.228 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:35:39.469 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:35:39.469 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:39.469 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:39.469 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:35:39.469 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:39.470 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:39.660 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T19:35:40.240 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:40.374 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:40.375 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:40.375 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-10T19:35:40.375 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:40.375 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:40.375 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 145 k 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 432 k 2024-09-10T19:35:40.376 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:40.561 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 788 kB/s | 145 kB 00:00 2024-09-10T19:35:40.561 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:40.562 INFO:teuthology.orchestra.run.smithi130.stdout:Total 781 kB/s | 145 kB 00:00 2024-09-10T19:35:40.562 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:40.572 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:40.572 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:40.627 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:40.627 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:40.776 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:40.854 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:35:40.879 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:35:40.879 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:35:40.879 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T19:35:40.879 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:41.843 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T19:35:41.843 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:41.843 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:41.843 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:35:41.844 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:41.844 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:42.057 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr 2024-09-10T19:35:42.629 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:42.721 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:35:42.763 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:42.764 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:35:42.764 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:42.824 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T19:35:43.380 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:43.513 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T19:35:43.514 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:43.515 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:35:43.516 INFO:teuthology.orchestra.run.smithi130.stdout:Install 12 Packages 2024-09-10T19:35:43.516 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:43.516 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 19 M 2024-09-10T19:35:43.516 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 159 M 2024-09-10T19:35:43.516 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:43.865 INFO:teuthology.orchestra.run.smithi130.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 234 kB/s | 34 kB 00:00 2024-09-10T19:35:43.890 INFO:teuthology.orchestra.run.smithi130.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 96 kB/s | 16 kB 00:00 2024-09-10T19:35:44.057 INFO:teuthology.orchestra.run.smithi130.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 269 kB 00:00 2024-09-10T19:35:44.199 INFO:teuthology.orchestra.run.smithi130.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-09-10T19:35:44.258 INFO:teuthology.orchestra.run.smithi130.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.2 MB/s | 863 kB 00:00 2024-09-10T19:35:44.400 INFO:teuthology.orchestra.run.smithi130.stdout:(6/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f55 21 MB/s | 14 MB 00:00 2024-09-10T19:35:44.442 INFO:teuthology.orchestra.run.smithi130.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-09-10T19:35:44.475 INFO:teuthology.orchestra.run.smithi130.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 89 kB/s | 19 kB 00:00 2024-09-10T19:35:44.501 INFO:teuthology.orchestra.run.smithi130.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.6 MB/s | 144 kB 00:00 2024-09-10T19:35:44.526 INFO:teuthology.orchestra.run.smithi130.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 612 kB/s | 31 kB 00:00 2024-09-10T19:35:44.551 INFO:teuthology.orchestra.run.smithi130.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T19:35:44.626 INFO:teuthology.orchestra.run.smithi130.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 9.3 MB/s | 2.0 MB 00:00 2024-09-10T19:35:44.631 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:44.631 INFO:teuthology.orchestra.run.smithi130.stdout:Total 17 MB/s | 19 MB 00:01 2024-09-10T19:35:44.679 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:44.697 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:44.697 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:44.971 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:44.972 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:45.570 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:45.651 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T19:35:45.687 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T19:35:45.733 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T19:35:45.763 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T19:35:45.863 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T19:35:45.989 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T19:35:46.082 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T19:35:46.126 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T19:35:46.154 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T19:35:46.187 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T19:35:49.346 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T19:35:49.400 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T19:35:51.350 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T19:35:51.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T19:35:51.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T19:35:51.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T19:35:51.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T19:35:51.351 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T19:35:51.684 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:51.685 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:35:51.922 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T19:35:52.485 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:08 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:35:52.619 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T19:35:52.620 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================= 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout:Install 12 Packages 2024-09-10T19:35:52.621 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:35:52.622 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 38 M 2024-09-10T19:35:52.622 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 212 M 2024-09-10T19:35:52.622 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:35:52.998 INFO:teuthology.orchestra.run.smithi130.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-10T19:35:53.065 INFO:teuthology.orchestra.run.smithi130.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.0 MB/s | 801 kB 00:00 2024-09-10T19:35:53.090 INFO:teuthology.orchestra.run.smithi130.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-09-10T19:35:53.140 INFO:teuthology.orchestra.run.smithi130.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 364 kB/s | 18 kB 00:00 2024-09-10T19:35:53.198 INFO:teuthology.orchestra.run.smithi130.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-10T19:35:53.349 INFO:teuthology.orchestra.run.smithi130.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-09-10T19:35:53.399 INFO:teuthology.orchestra.run.smithi130.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-09-10T19:35:53.491 INFO:teuthology.orchestra.run.smithi130.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 17 MB/s | 4.9 MB 00:00 2024-09-10T19:35:53.541 INFO:teuthology.orchestra.run.smithi130.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-09-10T19:35:53.608 INFO:teuthology.orchestra.run.smithi130.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-09-10T19:35:53.875 INFO:teuthology.orchestra.run.smithi130.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-10T19:35:53.967 INFO:teuthology.orchestra.run.smithi130.stdout:(12/12): ceph-mgr-diskprediction-local-19.3.0-4 6.7 MB/s | 7.4 MB 00:01 2024-09-10T19:35:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:35:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-10T19:35:54.321 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:35:54.347 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:35:54.348 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:35:54.777 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:35:54.777 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:35:55.533 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:35:55.602 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T19:35:55.640 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:35:55.738 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:35:55.777 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T19:35:55.959 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T19:35:55.990 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T19:35:56.058 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T19:35:56.728 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T19:35:56.855 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T19:35:58.153 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:35:59.184 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T19:35:59.214 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T19:36:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T19:36:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T19:36:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:36:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:36:01.651 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:36:01.652 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T19:36:02.043 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T19:36:02.043 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:02.043 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:02.043 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:36:02.043 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:02.044 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:02.421 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-rook 2024-09-10T19:36:02.999 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:18 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:03.133 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:03.134 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:36:03.189 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:03.190 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:03.191 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2024-09-10T19:36:03.191 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:03.191 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 2.2 M 2024-09-10T19:36:03.191 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 26 M 2024-09-10T19:36:03.191 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:04.004 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 125 kB/s | 49 kB 00:00 2024-09-10T19:36:04.146 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 36 kB/s | 19 kB 00:00 2024-09-10T19:36:04.180 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-09-10T19:36:04.838 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 266 kB/s | 222 kB 00:00 2024-09-10T19:36:04.905 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 209 kB/s | 159 kB 00:00 2024-09-10T19:36:04.947 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 782 kB/s | 32 kB 00:00 2024-09-10T19:36:04.980 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 424 kB/s | 14 kB 00:00 2024-09-10T19:36:05.014 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 6.5 MB/s | 220 kB 00:00 2024-09-10T19:36:05.073 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 18 MB/s | 1.0 MB 00:00 2024-09-10T19:36:05.131 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 293 kB/s | 279 kB 00:00 2024-09-10T19:36:05.156 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 791 kB/s | 59 kB 00:00 2024-09-10T19:36:05.198 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-10T19:36:05.465 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 86 kB/s | 54 kB 00:00 2024-09-10T19:36:05.469 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:05.469 INFO:teuthology.orchestra.run.smithi130.stdout:Total 972 kB/s | 2.2 MB 00:02 2024-09-10T19:36:05.540 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:05.556 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:05.556 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:05.646 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:05.646 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:06.174 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:06.343 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T19:36:06.400 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T19:36:06.455 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T19:36:06.492 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T19:36:06.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T19:36:06.635 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T19:36:06.745 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T19:36:06.802 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T19:36:07.519 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T19:36:07.560 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T19:36:07.593 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T19:36:07.641 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T19:36:07.663 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T19:36:08.423 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T19:36:08.424 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T19:36:08.757 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:08.758 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:08.999 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T19:36:09.566 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:25 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:09.699 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:09.700 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:09.700 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:36:09.700 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:09.700 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:09.700 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T19:36:09.701 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T19:36:09.702 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:09.702 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:09.702 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:09.702 INFO:teuthology.orchestra.run.smithi130.stdout:Install 6 Packages 2024-09-10T19:36:09.702 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:09.703 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 7.0 M 2024-09-10T19:36:09.703 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 33 M 2024-09-10T19:36:09.704 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:10.085 INFO:teuthology.orchestra.run.smithi130.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-09-10T19:36:10.110 INFO:teuthology.orchestra.run.smithi130.stdout:(2/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 622 kB/s | 154 kB 00:00 2024-09-10T19:36:10.169 INFO:teuthology.orchestra.run.smithi130.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 987 kB/s | 58 kB 00:00 2024-09-10T19:36:10.203 INFO:teuthology.orchestra.run.smithi130.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.2 MB/s | 74 kB 00:00 2024-09-10T19:36:10.236 INFO:teuthology.orchestra.run.smithi130.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 3.6 MB/s | 548 kB 00:00 2024-09-10T19:36:10.403 INFO:teuthology.orchestra.run.smithi130.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-09-10T19:36:10.405 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:10.406 INFO:teuthology.orchestra.run.smithi130.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-09-10T19:36:10.477 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:10.487 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:10.488 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:10.589 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:10.589 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:10.896 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:11.062 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T19:36:11.121 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T19:36:11.642 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T19:36:11.724 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T19:36:11.791 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T19:36:11.817 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T19:36:12.565 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T19:36:12.847 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T19:36:12.847 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:12.848 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:13.103 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-fuse 2024-09-10T19:36:13.669 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:29 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T19:36:13.802 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout:Install 2 Packages 2024-09-10T19:36:13.803 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:13.804 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 981 k 2024-09-10T19:36:13.804 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.9 M 2024-09-10T19:36:13.804 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:14.163 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 808 kB/s | 80 kB 00:00 2024-09-10T19:36:14.363 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 2.9 MB/s | 901 kB 00:00 2024-09-10T19:36:14.363 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:14.364 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.7 MB/s | 981 kB 00:00 2024-09-10T19:36:14.376 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:14.398 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:14.398 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:14.479 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:14.479 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:14.663 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:14.798 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T19:36:14.870 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T19:36:15.533 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T19:36:15.533 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T19:36:15.806 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T19:36:15.806 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:15.806 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:15.806 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T19:36:15.807 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:15.807 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:16.049 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-volume 2024-09-10T19:36:16.631 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:16.771 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:16.771 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:16.771 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-09-10T19:36:16.771 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:16.771 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 285 k 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.3 M 2024-09-10T19:36:16.772 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:16.991 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.3 MB/s | 285 kB 00:00 2024-09-10T19:36:16.992 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:16.992 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-10T19:36:16.992 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:17.000 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:17.000 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:17.034 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:17.035 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:17.256 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:17.318 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:36:17.334 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:36:17.334 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:36:17.334 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T19:36:17.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:18.345 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T19:36:18.345 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:18.345 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:18.345 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T19:36:18.345 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:18.346 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:18.612 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados-devel 2024-09-10T19:36:19.199 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:19.335 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:19.335 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 126 k 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 456 k 2024-09-10T19:36:19.336 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:19.546 INFO:teuthology.orchestra.run.smithi130.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 605 kB/s | 126 kB 00:00 2024-09-10T19:36:19.546 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:19.546 INFO:teuthology.orchestra.run.smithi130.stdout:Total 601 kB/s | 126 kB 00:00 2024-09-10T19:36:19.547 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:19.552 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:19.552 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:19.616 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:19.616 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:19.770 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:19.860 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:20.319 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:20.571 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:20.571 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:20.571 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:20.572 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:36:20.572 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:20.572 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:20.779 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs2 2024-09-10T19:36:21.357 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:21.448 INFO:teuthology.orchestra.run.smithi130.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:21.492 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:21.493 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:21.493 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:21.556 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs-devel 2024-09-10T19:36:22.123 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:22.255 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:22.256 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:22.256 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-10T19:36:22.256 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:22.256 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:22.256 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T19:36:22.257 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:22.257 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:22.257 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:22.257 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:22.257 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:22.258 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 32 k 2024-09-10T19:36:22.258 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 142 k 2024-09-10T19:36:22.258 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:22.398 INFO:teuthology.orchestra.run.smithi130.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 226 kB/s | 32 kB 00:00 2024-09-10T19:36:22.398 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:22.398 INFO:teuthology.orchestra.run.smithi130.stdout:Total 224 kB/s | 32 kB 00:00 2024-09-10T19:36:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:22.402 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:22.402 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:22.425 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:22.425 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:22.518 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:22.612 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:23.014 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:23.245 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:23.453 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados2 2024-09-10T19:36:24.015 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:24.106 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:24.149 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:24.150 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:24.150 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:24.209 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librbd1 2024-09-10T19:36:24.777 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:24.866 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:24.908 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:24.909 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:24.909 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:24.975 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rados 2024-09-10T19:36:25.546 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:25.637 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:25.681 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:25.681 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:25.682 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:25.739 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rgw 2024-09-10T19:36:26.331 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:26.428 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:26.473 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:26.474 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:26.474 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:26.536 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-cephfs 2024-09-10T19:36:27.118 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:27.209 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:27.253 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:27.254 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:27.254 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:27.315 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rbd 2024-09-10T19:36:27.886 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:27.976 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T19:36:28.019 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-09-10T19:36:28.020 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:28.083 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-fuse 2024-09-10T19:36:28.646 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:28.778 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:28.779 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:28.780 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 85 k 2024-09-10T19:36:28.780 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 231 k 2024-09-10T19:36:28.780 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:29.068 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 295 kB/s | 85 kB 00:00 2024-09-10T19:36:29.068 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:29.069 INFO:teuthology.orchestra.run.smithi130.stdout:Total 294 kB/s | 85 kB 00:00 2024-09-10T19:36:29.069 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:29.076 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:29.076 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:29.139 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:29.139 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:29.288 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:29.398 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:29.800 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:30.070 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:30.321 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-mirror 2024-09-10T19:36:30.906 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T19:36:31.044 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.2 M 2024-09-10T19:36:31.045 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 14 M 2024-09-10T19:36:31.046 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:32.262 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 2.6 MB/s | 3.2 MB 00:01 2024-09-10T19:36:32.263 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:32.264 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.6 MB/s | 3.2 MB 00:01 2024-09-10T19:36:32.264 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:32.275 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:32.275 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:32.366 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:32.366 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:32.809 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:32.880 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:32.905 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:32.905 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:36:32.906 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T19:36:32.906 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:36:32.906 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:36:32.906 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:34.004 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:34.232 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-nbd 2024-09-10T19:36:34.808 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Sep 2024 07:34:44 PM UTC. 2024-09-10T19:36:34.943 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-09-10T19:36:34.943 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 170 k 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 490 k 2024-09-10T19:36:34.944 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-09-10T19:36:35.157 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 805 kB/s | 170 kB 00:00 2024-09-10T19:36:35.157 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:36:35.158 INFO:teuthology.orchestra.run.smithi130.stdout:Total 798 kB/s | 170 kB 00:00 2024-09-10T19:36:35.158 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-09-10T19:36:35.165 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-09-10T19:36:35.165 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-09-10T19:36:35.230 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-09-10T19:36:35.230 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-09-10T19:36:35.379 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-09-10T19:36:35.495 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:35.905 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-09-10T19:36:36.169 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-09-10T19:36:36.417 DEBUG:teuthology.parallel:result is None 2024-09-10T19:36:36.418 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:36:36.568 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T19:36:36.589 INFO:teuthology.orchestra.run.smithi130.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T19:36:36.590 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T19:36:36.590 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T19:36:36.592 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T19:36:36.592 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:36:36.593 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T19:36:36.655 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T19:36:36.655 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:36:36.655 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T19:36:36.731 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T19:36:36.795 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T19:36:36.795 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:36:36.795 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T19:36:36.861 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T19:36:36.926 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T19:36:36.927 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:36:36.927 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T19:36:36.991 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T19:36:37.056 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T19:36:37.075 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:36:37.075 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T19:36:37.075 INFO:tasks.workunit:timeout=3h 2024-09-10T19:36:37.076 INFO:tasks.workunit:cleanup=True 2024-09-10T19:36:37.076 DEBUG:teuthology.orchestra.run.smithi130:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:36:37.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:36:37.112 INFO:teuthology.orchestra.run.smithi130.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T19:36:37.112 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T19:36:37.167 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T19:36:37.167 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T19:36:37.223 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T19:36:37.280 INFO:tasks.workunit.client.0.smithi130.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.762 INFO:tasks.workunit.client.0.smithi130.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: git switch -c 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:Or undo this operation with: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: git switch - 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:37:25.763 INFO:tasks.workunit.client.0.smithi130.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T19:37:25.765 DEBUG:teuthology.orchestra.run.smithi130:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T19:37:25.822 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-09-10T19:37:25.822 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T19:37:25.878 INFO:tasks.workunit:Running workunits matching osd-backfill on client.0... 2024-09-10T19:37:25.879 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-prio.sh... 2024-09-10T19:37:25.879 DEBUG:teuthology.orchestra.run.smithi130:workunit test osd-backfill/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh 2024-09-10T19:37:25.944 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T19:37:25.949 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T19:37:25.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-backfill-prio 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:22: run: shift 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-09-10T19:37:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2024-09-10T19:37:25.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: uuidgen 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none ' 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:32: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: export objects=50 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:33: run: objects=50 2024-09-10T19:37:25.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: export poolprefix=test 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:34: run: poolprefix=test 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: export FORCE_PRIO=254 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:35: run: FORCE_PRIO=254 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: export DEGRADED_PRIO=150 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:36: run: DEGRADED_PRIO=150 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: export NORMAL_PRIO=110 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:37: run: NORMAL_PRIO=110 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: set 2024-09-10T19:37:25.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:39: run: local 'funcs=TEST_backfill_pool_priority 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_priority' 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-prio 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-prio 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-prio 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-prio KILL 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:37:25.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:37:25.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:37:25.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:37:25.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:37:25.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:37:25.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:37:25.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:37:25.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:37:25.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:37:25.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:37:25.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:37:25.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:37:25.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-prio 2024-09-10T19:37:25.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:37:25.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:25.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:25.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-10T19:37:25.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:37:25.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:37:25.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-prio 2024-09-10T19:37:25.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:37:25.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:25.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:25.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27247 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_pool_priority td/osd-backfill-prio 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:329: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: local pools=3 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: local OSDS=2 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:333: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-prio 2024-09-10T19:37:25.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:37:25.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:37:25.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:37:25.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-prio/a 2024-09-10T19:37:25.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2024-09-10T19:37:26.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:37:26.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:37:26.485 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:37:26.485 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:37:26.486 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:26.486 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:26.486 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:26.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:37:26.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:37:26.679 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:37:26.680 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:26.680 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:26.680 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-10T19:37:26.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:37:26.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get fsid 2024-09-10T19:37:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:37:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:37:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:37:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:37:26.764 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:37:26.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get mon_host 2024-09-10T19:37:26.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2024-09-10T19:37:26.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-prio 2024-09-10T19:37:26.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:37:26.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:37:26.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:37:26.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-prio/x 2024-09-10T19:37:26.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:37:27.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:37:27.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:37:27.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:37:27.691 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:37:27.691 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:27.692 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:27.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:27.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:37:27.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:37:27.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:335: TEST_backfill_pool_priority: export CEPH_ARGS 2024-09-10T19:37:27.724 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: expr 2 - 1 2024-09-10T19:37:27.725 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: seq 0 1 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/0 2024-09-10T19:37:27.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:27.739 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:37:27.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/0 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5c566648-9eee-4d0d-86a3-5d7cf7c55090 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 5c566648-9eee-4d0d-86a3-5d7cf7c55090 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5c566648-9eee-4d0d-86a3-5d7cf7c55090' 2024-09-10T19:37:27.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:37:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD3n+BmPi9MLRAADzzQXzj79ESHQO+aTfBGBg== 2024-09-10T19:37:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD3n+BmPi9MLRAADzzQXzj79ESHQO+aTfBGBg=="}' 2024-09-10T19:37:27.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5c566648-9eee-4d0d-86a3-5d7cf7c55090 -i td/osd-backfill-prio/0/new.json 2024-09-10T19:37:28.036 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:37:28.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/0/new.json 2024-09-10T19:37:28.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3n+BmPi9MLRAADzzQXzj79ESHQO+aTfBGBg== --osd-uuid 5c566648-9eee-4d0d-86a3-5d7cf7c55090 2024-09-10T19:37:28.111 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:28.109+0000 7fe24d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:28.113 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:28.111+0000 7fe24d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:28.114 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:28.112+0000 7fe24d413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:28.114 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:28.113+0000 7fe24d413640 -1 bdev(0x560cde497c00 td/osd-backfill-prio/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:37:28.115 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:28.113+0000 7fe24d413640 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2024-09-10T19:37:33.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2024-09-10T19:37:33.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:37:33.609 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:37:33.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:37:33.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:37:34.137 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:37:34.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:37:34.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:37:34.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:37:34.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:37:34.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:37:34.171 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:34.170+0000 7ffbf0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:34.190 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:34.189+0000 7ffbf0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:34.192 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:34.190+0000 7ffbf0e19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:34.596 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:37:34.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:37:34.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:34.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:37:35.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:35.537 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:35.536+0000 7ffbf0e19640 -1 Falling back to public interface 2024-09-10T19:37:36.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:36.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:36.024 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:37:36.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:37:36.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:36.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:37:36.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:37.369 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:37.367+0000 7ffbf0e19640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:37:37.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:37.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:37.450 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:37:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:37:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:37.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:37:37.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:38.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:38.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:38.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:37:38.980 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:37:38.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:38.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:37:39.373 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2125776286,v1:127.0.0.1:6803/2125776286] [v2:127.0.0.1:6804/2125776286,v1:127.0.0.1:6805/2125776286] exists,up 5c566648-9eee-4d0d-86a3-5d7cf7c55090 2024-09-10T19:37:39.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:37:39.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:37:39.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:37:39.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:337: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/1 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:37:39.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:37:39.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/1 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d2720def-c2c5-4e96-af50-54d454bd3df4 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d2720def-c2c5-4e96-af50-54d454bd3df4' 2024-09-10T19:37:39.380 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 d2720def-c2c5-4e96-af50-54d454bd3df4 2024-09-10T19:37:39.381 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:37:39.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADoOBmzUe/FxAAcjZahvnCEjthVK/o8wRyCQ== 2024-09-10T19:37:39.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADoOBmzUe/FxAAcjZahvnCEjthVK/o8wRyCQ=="}' 2024-09-10T19:37:39.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d2720def-c2c5-4e96-af50-54d454bd3df4 -i td/osd-backfill-prio/1/new.json 2024-09-10T19:37:39.857 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:37:39.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/1/new.json 2024-09-10T19:37:39.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADoOBmzUe/FxAAcjZahvnCEjthVK/o8wRyCQ== --osd-uuid d2720def-c2c5-4e96-af50-54d454bd3df4 2024-09-10T19:37:39.904 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:39.901+0000 7f86e380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:39.917 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:39.915+0000 7f86e380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:39.918 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:39.916+0000 7f86e380b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:39.919 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:39.916+0000 7f86e380b640 -1 bdev(0x56472efd7c00 td/osd-backfill-prio/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:37:39.919 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:39.916+0000 7f86e380b640 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2024-09-10T19:37:45.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2024-09-10T19:37:45.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:37:45.536 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:37:45.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:37:45.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:37:46.086 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:37:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:37:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:37:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:37:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:37:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:37:46.120 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:46.117+0000 7f3e9d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:46.122 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:46.120+0000 7f3e9d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:46.123 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:46.121+0000 7f3e9d00d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:46.509 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:37:46.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:37:46.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:46.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:37:46.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:47.467 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:47.464+0000 7f3e9d00d640 -1 Falling back to public interface 2024-09-10T19:37:47.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:47.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:47.911 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:37:47.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:37:47.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:47.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:37:48.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:49.265 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:37:49.264+0000 7f3e9d00d640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:37:49.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:49.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:49.330 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:37:49.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:37:49.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:49.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:37:49.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:37:50.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:37:50.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:37:50.820 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:37:50.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:37:50.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:37:50.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:37:51.246 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4102949053,v1:127.0.0.1:6811/4102949053] [v2:127.0.0.1:6812/4102949053,v1:127.0.0.1:6813/4102949053] exists,up d2720def-c2c5-4e96-af50-54d454bd3df4 2024-09-10T19:37:51.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:37:51.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:37:51.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:37:51.248 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: seq 1 3 2024-09-10T19:37:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:37:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test1 1 1 2024-09-10T19:37:51.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test1 1 1 2024-09-10T19:37:51.803 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' already exists 2024-09-10T19:37:51.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:37:52.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2024-09-10T19:37:54.142 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2024-09-10T19:37:54.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:37:54.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test2 1 1 2024-09-10T19:37:54.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test2 1 1 2024-09-10T19:37:54.766 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' already exists 2024-09-10T19:37:54.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:37:55.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2024-09-10T19:37:56.723 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2024-09-10T19:37:56.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:37:56.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: create_pool test3 1 1 2024-09-10T19:37:56.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test3 1 1 2024-09-10T19:37:57.310 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' already exists 2024-09-10T19:37:57.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:37:58.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:345: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2024-09-10T19:37:59.018 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2024-09-10T19:37:59.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:347: TEST_backfill_pool_priority: sleep 5 2024-09-10T19:38:04.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:349: TEST_backfill_pool_priority: wait_for_clean 2024-09-10T19:38:04.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:38:04.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:38:04.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:38:04.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:38:04.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:38:04.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:38:04.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:38:04.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:38:04.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:38:04.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:38:04.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:38:04.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:38:04.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:38:04.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:38:04.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:38:04.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:38:04.549 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-09-10T19:38:04.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:38:04.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:38:04.549 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:38:04.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836488 2024-09-10T19:38:04.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836488 2024-09-10T19:38:04.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488' 2024-09-10T19:38:04.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:38:04.667 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:38:04.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672965 2024-09-10T19:38:04.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672965 2024-09-10T19:38:04.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836488 1-42949672965' 2024-09-10T19:38:04.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:38:04.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836488 2024-09-10T19:38:04.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:38:04.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:38:04.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836488 2024-09-10T19:38:04.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:38:04.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836488 2024-09-10T19:38:04.789 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836488 2024-09-10T19:38:04.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-10T19:38:04.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:38:05.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836485 -lt 21474836488 2024-09-10T19:38:05.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:38:06.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:38:06.217 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:38:06.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-10T19:38:06.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:38:06.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672965 2024-09-10T19:38:06.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:38:06.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:38:06.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672965 2024-09-10T19:38:06.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:38:06.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672965 2024-09-10T19:38:06.655 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672965 2024-09-10T19:38:06.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-10T19:38:06.656 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:38:07.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672965 -lt 42949672965 2024-09-10T19:38:07.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:38:07.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:38:07.120 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:38:07.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:38:08.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T19:38:08.031 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:38:08.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:38:08.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:38:08.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-10T19:38:08.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:38:08.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:38:08.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:351: TEST_backfill_pool_priority: ceph pg dump pgs 2024-09-10T19:38:08.938 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:38:08.938 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:38:02.024292+0000 0'0 28:19 [1,0] 1 [1,0] 1 0'0 2024-09-10T19:37:57.136318+0000 0'0 2024-09-10T19:37:57.136318+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:52:43.413381+0000 0 0 2024-09-10T19:38:08.938 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:37:56.949481+0000 0'0 28:33 [0,1] 0 [0,1] 0 0'0 2024-09-10T19:37:54.486048+0000 0'0 2024-09-10T19:37:54.486048+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:04:08.365643+0000 0 0 2024-09-10T19:38:08.938 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:37:54.110006+0000 0'0 28:39 [1,0] 1 [1,0] 1 0'0 2024-09-10T19:37:51.558841+0000 0'0 2024-09-10T19:37:51.558841+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:56:09.586952+0000 0 0 2024-09-10T19:38:08.938 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:38:08.939 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:38:08.939 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:38:08.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local PG1 2024-09-10T19:38:08.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local POOLNUM1 2024-09-10T19:38:08.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:357: TEST_backfill_pool_priority: local pool1 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local chk_osd1_1 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local chk_osd1_2 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local PG2 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local POOLNUM2 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:363: TEST_backfill_pool_priority: local pool2 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: local chk_osd2_1 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:365: TEST_backfill_pool_priority: local chk_osd2_2 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: seq 1 3 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2024-09-10T19:38:08.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2024-09-10T19:38:09.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:38:09.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=1 2024-09-10T19:38:09.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:38:09.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=0 2024-09-10T19:38:09.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z '' ']' 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: PG1=1.0 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: POOLNUM1=1 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: pool1=test1 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:377: TEST_backfill_pool_priority: chk_osd1_1=1 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: chk_osd1_2=0 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2024-09-10T19:38:09.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: jq '.acting[]' 2024-09-10T19:38:09.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:38:09.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:370: TEST_backfill_pool_priority: local test_osd1=0 2024-09-10T19:38:09.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: local test_osd2=1 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: PG2=2.0 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: POOLNUM2=2 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: pool2=test2 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:384: TEST_backfill_pool_priority: chk_osd2_1=0 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:385: TEST_backfill_pool_priority: chk_osd2_2=1 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: break 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:389: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2024-09-10T19:38:09.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:391: TEST_backfill_pool_priority: '[' test2 = '' ']' 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: seq 1 3 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:397: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:399: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:401: TEST_backfill_pool_priority: delete_pool test3 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test3 2024-09-10T19:38:09.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-10T19:38:10.278 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' does not exist 2024-09-10T19:38:10.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_extra_prio=1 2024-09-10T19:38:10.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_extra_prio=2 2024-09-10T19:38:10.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: expr 150 + 1 + 1 2024-09-10T19:38:10.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: pool1_prio=152 2024-09-10T19:38:10.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: expr 150 + 1 + 2 2024-09-10T19:38:10.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: pool2_prio=153 2024-09-10T19:38:10.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-10T19:38:11.035 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 1 2024-09-10T19:38:11.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-09-10T19:38:11.806 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 recovery_priority to 1 2024-09-10T19:38:11.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:413: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-09-10T19:38:12.718 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 1 2024-09-10T19:38:12.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-09-10T19:38:13.431 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 recovery_priority to 2 2024-09-10T19:38:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: wait_for_clean 2024-09-10T19:38:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:38:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:38:13.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:38:13.443 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:38:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:38:13.443 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:38:13.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:38:13.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:38:13.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:38:13.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:38:13.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:38:13.940 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-09-10T19:38:13.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:38:13.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:38:13.940 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:38:14.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836492 2024-09-10T19:38:14.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836492 2024-09-10T19:38:14.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492' 2024-09-10T19:38:14.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:38:14.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672969 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672969 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836492 2024-09-10T19:38:14.221 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:38:14.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:38:14.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836492 2024-09-10T19:38:14.223 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:38:14.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836492 2024-09-10T19:38:14.224 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836492 2024-09-10T19:38:14.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-10T19:38:14.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:38:14.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836489 -lt 21474836492 2024-09-10T19:38:14.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:38:15.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:38:15.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:38:16.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-10T19:38:16.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:38:16.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672969 2024-09-10T19:38:16.052 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:38:16.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:38:16.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672969 2024-09-10T19:38:16.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:38:16.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672969 2024-09-10T19:38:16.055 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672969 2024-09-10T19:38:16.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-10T19:38:16.056 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:38:16.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672969 -lt 42949672969 2024-09-10T19:38:16.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:38:16.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:38:16.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 2 == 0 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:38:17.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:38:17.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:38:17.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:38:17.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:38:17.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:38:18.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 2 2024-09-10T19:38:18.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:38:18.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:38:18.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:417: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2024-09-10T19:38:18.066 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records in 2024-09-10T19:38:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records out 2024-09-10T19:38:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0286286 s, 366 MB/s 2024-09-10T19:38:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: p=1 2024-09-10T19:38:18.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2024-09-10T19:38:18.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2024-09-10T19:38:18.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:18.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2024-09-10T19:38:18.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:18.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2024-09-10T19:38:18.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:18.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2024-09-10T19:38:19.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:19.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2024-09-10T19:38:19.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:19.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2024-09-10T19:38:19.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:19.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2024-09-10T19:38:20.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:20.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2024-09-10T19:38:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:20.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2024-09-10T19:38:20.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:20.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2024-09-10T19:38:21.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:21.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2024-09-10T19:38:21.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:21.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2024-09-10T19:38:21.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:21.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2024-09-10T19:38:22.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:22.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2024-09-10T19:38:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2024-09-10T19:38:22.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:22.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2024-09-10T19:38:23.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:23.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2024-09-10T19:38:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:23.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2024-09-10T19:38:24.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:24.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2024-09-10T19:38:24.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:24.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2024-09-10T19:38:24.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:24.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2024-09-10T19:38:25.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:25.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2024-09-10T19:38:25.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:25.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2024-09-10T19:38:25.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:25.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2024-09-10T19:38:26.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:26.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2024-09-10T19:38:26.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:26.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2024-09-10T19:38:26.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:26.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2024-09-10T19:38:27.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:27.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2024-09-10T19:38:27.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:27.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2024-09-10T19:38:28.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:28.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2024-09-10T19:38:28.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:28.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2024-09-10T19:38:28.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:28.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2024-09-10T19:38:29.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:29.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2024-09-10T19:38:29.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:29.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2024-09-10T19:38:29.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:29.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2024-09-10T19:38:30.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:30.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2024-09-10T19:38:30.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:30.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2024-09-10T19:38:30.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:30.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2024-09-10T19:38:31.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:31.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2024-09-10T19:38:31.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:31.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2024-09-10T19:38:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:32.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2024-09-10T19:38:32.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:32.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2024-09-10T19:38:32.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:32.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2024-09-10T19:38:33.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:33.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2024-09-10T19:38:33.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:33.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2024-09-10T19:38:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:34.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2024-09-10T19:38:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:34.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2024-09-10T19:38:34.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:34.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2024-09-10T19:38:35.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:35.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2024-09-10T19:38:35.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:35.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2024-09-10T19:38:35.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:35.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2024-09-10T19:38:36.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 1 + 1 2024-09-10T19:38:36.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=2 2024-09-10T19:38:36.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:419: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2024-09-10T19:38:36.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: seq 1 50 2024-09-10T19:38:36.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:36.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2024-09-10T19:38:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:36.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2024-09-10T19:38:36.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:36.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2024-09-10T19:38:37.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:37.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2024-09-10T19:38:37.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:37.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2024-09-10T19:38:38.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:38.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2024-09-10T19:38:38.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:38.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2024-09-10T19:38:38.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:38.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2024-09-10T19:38:39.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:39.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2024-09-10T19:38:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:39.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2024-09-10T19:38:39.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:39.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2024-09-10T19:38:40.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:40.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2024-09-10T19:38:40.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:40.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2024-09-10T19:38:40.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:40.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2024-09-10T19:38:41.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:41.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2024-09-10T19:38:41.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:41.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2024-09-10T19:38:42.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:42.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2024-09-10T19:38:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:42.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2024-09-10T19:38:42.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:42.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2024-09-10T19:38:43.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:43.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2024-09-10T19:38:43.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:43.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2024-09-10T19:38:43.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:43.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2024-09-10T19:38:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:44.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2024-09-10T19:38:44.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:44.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2024-09-10T19:38:44.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:44.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2024-09-10T19:38:45.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:45.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2024-09-10T19:38:45.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:45.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2024-09-10T19:38:46.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:46.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2024-09-10T19:38:46.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:46.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2024-09-10T19:38:46.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:46.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2024-09-10T19:38:47.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:47.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2024-09-10T19:38:47.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:47.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2024-09-10T19:38:47.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:47.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2024-09-10T19:38:48.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:48.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2024-09-10T19:38:48.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:48.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2024-09-10T19:38:48.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:48.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2024-09-10T19:38:49.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:49.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2024-09-10T19:38:49.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:49.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2024-09-10T19:38:50.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:50.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2024-09-10T19:38:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:50.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2024-09-10T19:38:50.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:50.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2024-09-10T19:38:51.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:51.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2024-09-10T19:38:51.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:51.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2024-09-10T19:38:51.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:51.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2024-09-10T19:38:52.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:52.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2024-09-10T19:38:52.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:52.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2024-09-10T19:38:52.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:52.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2024-09-10T19:38:53.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:53.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2024-09-10T19:38:53.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:53.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2024-09-10T19:38:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:421: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2024-09-10T19:38:53.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:423: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2024-09-10T19:38:54.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: expr 2 + 1 2024-09-10T19:38:54.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: p=3 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test1 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1-p1 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test1 obj1-p1 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test1 2024-09-10T19:38:54.172 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1-p1 2024-09-10T19:38:54.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-10T19:38:54.173 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T19:38:54.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T19:38:54.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-10T19:38:54.622 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T19:38:55.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: local otherosd=null 2024-09-10T19:38:55.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph pg dump pgs 2024-09-10T19:38:55.456 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:38:55.456 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 49 0 0 0 0 507510784 0 0 45 100 45 active+clean 2024-09-10T19:38:12.893601+0000 38'145 38:202 [0] 0 [0] 0 0'0 2024-09-10T19:37:54.486048+0000 0'0 2024-09-10T19:37:54.486048+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:41:11.299189+0000 0 0 2024-09-10T19:38:55.456 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:38:11.469575+0000 38'150 38:213 [1] 1 [1] 1 0'0 2024-09-10T19:37:51.558841+0000 0'0 2024-09-10T19:37:51.558841+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:48:13.529623+0000 0 0 2024-09-10T19:38:55.456 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:38:55.456 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:38:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:38:55.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ERRORS=0 2024-09-10T19:38:55.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2024-09-10T19:38:56.112 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2024-09-10T19:38:56.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2024-09-10T19:38:57.211 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2024-09-10T19:38:57.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: sleep 5 2024-09-10T19:39:02.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.1 2024-09-10T19:39:02.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:39:02.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:39:02.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:39:02.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:02.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:39:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2024-09-10T19:39:02.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:39:02.319 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 2024-09-10T19:39:02.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.1 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:39:02.320 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 152, 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "2.0", 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 153, 2024-09-10T19:39:02.321 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:39:02.322 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.322 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:39:02.322 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.322 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: get_asok_path osd.0 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.0.asok 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: CEPH_ARGS= 2024-09-10T19:39:02.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:439: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.0.asok dump_recovery_reservations 2024-09-10T19:39:02.408 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 2024-09-10T19:39:02.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:440: TEST_backfill_pool_priority: echo osd.0 2024-09-10T19:39:02.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2024-09-10T19:39:02.408 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "2.0", 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 153, 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:39:02.409 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 152, 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:39:02.410 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:39:02.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2024-09-10T19:39:02.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-10T19:39:02.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2024-09-10T19:39:02.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:444: TEST_backfill_pool_priority: ITEM=1.0 2024-09-10T19:39:02.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:445: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-10T19:39:02.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2024-09-10T19:39:02.413 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-10T19:39:02.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:450: TEST_backfill_pool_priority: PRIO=152 2024-09-10T19:39:02.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:451: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2024-09-10T19:39:02.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2024-09-10T19:39:02.416 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-10T19:39:02.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2024-09-10T19:39:02.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:459: TEST_backfill_pool_priority: ITEM=1.0 2024-09-10T19:39:02.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:460: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-10T19:39:02.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2024-09-10T19:39:02.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-10T19:39:02.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:465: TEST_backfill_pool_priority: PRIO=152 2024-09-10T19:39:02.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:466: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2024-09-10T19:39:02.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2024-09-10T19:39:02.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-10T19:39:02.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2024-09-10T19:39:02.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:474: TEST_backfill_pool_priority: ITEM=2.0 2024-09-10T19:39:02.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:475: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-10T19:39:02.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2024-09-10T19:39:02.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-10T19:39:02.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:480: TEST_backfill_pool_priority: PRIO=153 2024-09-10T19:39:02.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:481: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2024-09-10T19:39:02.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2024-09-10T19:39:02.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-10T19:39:02.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2024-09-10T19:39:02.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:489: TEST_backfill_pool_priority: ITEM=2.0 2024-09-10T19:39:02.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:490: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-10T19:39:02.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2024-09-10T19:39:02.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-10T19:39:02.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:495: TEST_backfill_pool_priority: PRIO=153 2024-09-10T19:39:02.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:496: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2024-09-10T19:39:02.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:503: TEST_backfill_pool_priority: wait_for_clean 2024-09-10T19:39:02.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:39:02.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:39:02.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:39:02.435 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:39:02.435 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:39:02.435 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:39:02.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:39:02.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:39:02.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:39:02.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:39:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:39:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:39:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:39:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:39:02.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:39:02.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:39:02.992 INFO:tasks.workunit.client.0.smithi130.stderr:1' 2024-09-10T19:39:02.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:39:02.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:39:02.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:39:03.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836503 2024-09-10T19:39:03.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836503 2024-09-10T19:39:03.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503' 2024-09-10T19:39:03.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:39:03.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:39:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672981 2024-09-10T19:39:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672981 2024-09-10T19:39:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836503 1-42949672981' 2024-09-10T19:39:03.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:39:03.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836503 2024-09-10T19:39:03.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:39:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:39:03.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836503 2024-09-10T19:39:03.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:39:03.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836503 2024-09-10T19:39:03.317 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836503 2024-09-10T19:39:03.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-10T19:39:03.318 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:39:03.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-10T19:39:03.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:39:03.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672981 2024-09-10T19:39:03.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:39:03.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:39:03.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672981 2024-09-10T19:39:03.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:39:03.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672981 2024-09-10T19:39:03.882 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672981 2024-09-10T19:39:03.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-10T19:39:03.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:39:04.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-10T19:39:04.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:39:04.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:04.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:05.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 2 == 0 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:05.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:05.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:05.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:05.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:05.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:06.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:06.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T19:39:06.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:39:06.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T19:39:06.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:06.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:06.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:07.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:07.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:07.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:07.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:07.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:07.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:08.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=18870294 2024-09-10T19:39:08.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 18870294 '!=' null 2024-09-10T19:39:08.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:08.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:08.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:08.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:08.464 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:08.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:08.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:08.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:08.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:08.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:08.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:09.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:09.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:09.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:09.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=22657601 2024-09-10T19:39:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 22657601 '!=' null 2024-09-10T19:39:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:10.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:10.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:10.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:10.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:10.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:10.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:10.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:10.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:11.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=27956055 2024-09-10T19:39:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 27956055 '!=' null 2024-09-10T19:39:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:12.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:12.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:12.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:12.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:12.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:12.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:12.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:12.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:13.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:13.555 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:13.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:14.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23588304 2024-09-10T19:39:14.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23588304 '!=' null 2024-09-10T19:39:14.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:14.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:14.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:14.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:14.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:14.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:14.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:14.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:14.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:14.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:15.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:15.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:15.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:15.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:15.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:16.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=21841487 2024-09-10T19:39:16.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 21841487 '!=' null 2024-09-10T19:39:16.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:16.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:16.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:16.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:16.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:16.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:16.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:16.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:16.776 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:16.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:17.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:17.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:17.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:17.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:18.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=27083312 2024-09-10T19:39:18.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 27083312 '!=' null 2024-09-10T19:39:18.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:18.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:18.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:18.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:18.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:18.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:18.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:18.875 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:19.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:19.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:19.625 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=29704430 2024-09-10T19:39:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 29704430 '!=' null 2024-09-10T19:39:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:20.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:20.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:20.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:20.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:20.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:20.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:20.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:20.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:20.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:21.206 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:21.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=29704595 2024-09-10T19:39:21.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 29704595 '!=' null 2024-09-10T19:39:21.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:21.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:21.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:22.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:22.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:22.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:22.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:22.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:22.447 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:22.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:23.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:23.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:23.104 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23588960 2024-09-10T19:39:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23588960 '!=' null 2024-09-10T19:39:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:24.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:24.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:24.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:24.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:24.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:24.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:24.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:25.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:26.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=29704146 2024-09-10T19:39:26.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 29704146 '!=' null 2024-09-10T19:39:26.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:26.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:26.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:26.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:26.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:26.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:26.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:26.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:26.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:26.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:26.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:26.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:26.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:26.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:27.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:27.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:28.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=28830097 2024-09-10T19:39:28.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 28830097 '!=' null 2024-09-10T19:39:28.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:28.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:28.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:28.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:28.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:28.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:28.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:28.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:28.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:28.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:29.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:29.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:29.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:30.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=28830225 2024-09-10T19:39:30.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 28830225 '!=' null 2024-09-10T19:39:30.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:30.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:30.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:30.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:30.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:30.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:30.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:30.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:30.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:30.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:30.157 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:30.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:30.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:30.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:30.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:31.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:31.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=29703483 2024-09-10T19:39:31.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 29703483 '!=' null 2024-09-10T19:39:31.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:31.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:31.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:31.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:31.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:31.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:32.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:32.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:32.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:32.252 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:32.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:32.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:32.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:32.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:32.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:32.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:32.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:32.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:32.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=29703483 2024-09-10T19:39:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 29703483 '!=' null 2024-09-10T19:39:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:33.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:33.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:33.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:33.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:33.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:33.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:33.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:33.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:34.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:34.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:35.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23588130 2024-09-10T19:39:35.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23588130 '!=' null 2024-09-10T19:39:35.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:35.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:35.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:35.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:35.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:35.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:35.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:35.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:35.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:35.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:35.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:35.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:35.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:35.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:35.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:36.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:36.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:36.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:36.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:36.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:36.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:36.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:36.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:36.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:37.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=27258030 2024-09-10T19:39:37.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 27258030 '!=' null 2024-09-10T19:39:37.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:37.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:37.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:37.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:37.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:37.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:37.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:37.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:37.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:37.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:38.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=28831533 2024-09-10T19:39:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 28831533 '!=' null 2024-09-10T19:39:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:38.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:38.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:38.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:38.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:38.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:38.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:38.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:38.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:38.907 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:39.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:39:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:39.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 2 2024-09-10T19:39:39.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:39:39.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:39:39.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:39:39.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:39:39.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:39:39.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:39:39.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:39:39.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:39:40.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=19013497 2024-09-10T19:39:40.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 19013497 '!=' null 2024-09-10T19:39:40.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:39:40.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:40.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:40.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:40.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:40.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:40.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:39:40.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:40.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:40.846 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:41.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 2 2024-09-10T19:39:41.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 0 2024-09-10T19:39:41.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:39:41.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-10T19:39:41.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:39:41.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:39:41.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:39:41.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:39:41.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:39:41.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:39:41.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:39:41.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:39:41.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:39:41.476 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:39:41.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:39:41.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:39:41.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:39:41.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:39:42.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 2 2024-09-10T19:39:42.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:39:42.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:39:42.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:505: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2024-09-10T19:39:42.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: echo TEST PASSED 2024-09-10T19:39:42.374 INFO:tasks.workunit.client.0.smithi130.stdout:TEST PASSED 2024-09-10T19:39:42.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: delete_pool test1 2024-09-10T19:39:42.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test1 2024-09-10T19:39:42.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-10T19:39:42.866 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' does not exist 2024-09-10T19:39:42.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:513: TEST_backfill_pool_priority: delete_pool test2 2024-09-10T19:39:42.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test2 2024-09-10T19:39:42.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-10T19:39:43.464 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' does not exist 2024-09-10T19:39:43.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:514: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2024-09-10T19:39:43.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:39:43.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:39:43.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:39:43.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:39:43.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:39:44.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:39:44.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:515: TEST_backfill_pool_priority: return 0 2024-09-10T19:39:44.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:43: run: teardown td/osd-backfill-prio 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-prio 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-prio KILL 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:39:44.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:39:44.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:39:44.792 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:39:44.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:39:44.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:39:44.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:39:44.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:39:44.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:39:44.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:39:44.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:39:44.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:39:44.796 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:39:44.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:39:44.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:39:44.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-prio 2024-09-10T19:39:44.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:39:44.814 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:44.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:44.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-10T19:39:44.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:39:44.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:39:44.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:40: run: for func in $funcs 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:41: run: setup td/osd-backfill-prio 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-prio 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-prio 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-prio 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-prio KILL 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:39:44.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:39:44.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:39:44.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:39:44.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:39:44.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:39:44.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:39:44.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:39:44.821 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:39:44.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:39:44.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:39:44.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:39:44.822 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:39:44.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:39:44.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:39:44.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-prio 2024-09-10T19:39:44.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:39:44.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:44.826 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:44.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-10T19:39:44.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:39:44.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:39:44.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-prio 2024-09-10T19:39:44.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:39:44.828 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:44.829 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:44.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.27247 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:42: run: TEST_backfill_priority td/osd-backfill-prio 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:49: TEST_backfill_priority: local dir=td/osd-backfill-prio 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:50: TEST_backfill_priority: local pools=10 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:51: TEST_backfill_priority: local OSDS=5 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: expr 150 + 1 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:53: TEST_backfill_priority: local degraded_prio=151 2024-09-10T19:39:44.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:54: TEST_backfill_priority: local max_tries=10 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:56: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-prio 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-prio/a 2024-09-10T19:39:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:45.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:39:45.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:45.490 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-10T19:39:45.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:39:45.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get fsid 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:39:45.574 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-mon.a.asok 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:39:45.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.27247/ceph-mon.a.asok config get mon_host 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:57: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-prio 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:39:45.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-prio/x 2024-09-10T19:39:45.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:46.516 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:46.517 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:39:46.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:39:46.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:58: TEST_backfill_priority: export CEPH_ARGS 2024-09-10T19:39:46.550 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: expr 5 - 1 2024-09-10T19:39:46.551 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: seq 0 4 2024-09-10T19:39:46.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:39:46.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2024-09-10T19:39:46.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/0 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:39:46.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:39:46.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/0 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:39:46.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=04642d86-93c8-4ec6-b3d3-29b9389207bb 2024-09-10T19:39:46.558 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 04642d86-93c8-4ec6-b3d3-29b9389207bb 2024-09-10T19:39:46.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 04642d86-93c8-4ec6-b3d3-29b9389207bb' 2024-09-10T19:39:46.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:39:46.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCCoOBmHeZZIhAAVKPK8iutf3eWH560xmCp/A== 2024-09-10T19:39:46.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCCoOBmHeZZIhAAVKPK8iutf3eWH560xmCp/A=="}' 2024-09-10T19:39:46.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 04642d86-93c8-4ec6-b3d3-29b9389207bb -i td/osd-backfill-prio/0/new.json 2024-09-10T19:39:46.836 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:39:46.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/0/new.json 2024-09-10T19:39:46.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCoOBmHeZZIhAAVKPK8iutf3eWH560xmCp/A== --osd-uuid 04642d86-93c8-4ec6-b3d3-29b9389207bb 2024-09-10T19:39:46.883 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:46.881+0000 7f609922e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:46.905 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:46.903+0000 7f609922e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:46.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:46.904+0000 7f609922e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:46.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:46.904+0000 7f609922e640 -1 bdev(0x5652de775c00 td/osd-backfill-prio/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:39:46.906 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:46.904+0000 7f609922e640 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2024-09-10T19:39:52.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2024-09-10T19:39:52.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:39:52.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:39:52.570 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:39:52.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:39:53.125 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:39:53.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:39:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:39:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:39:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:39:53.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:39:53.159 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:53.157+0000 7f9a3ee87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:53.161 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:53.159+0000 7f9a3ee87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:53.162 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:53.160+0000 7f9a3ee87640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:53.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:39:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:39:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:39:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:39:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:39:53.537 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:39:53.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:39:53.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:39:53.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:39:53.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:39:53.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:39:54.004 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:54.003+0000 7f9a3ee87640 -1 Falling back to public interface 2024-09-10T19:39:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:39:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:39:54.960 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:39:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:39:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:39:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:39:55.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:39:56.071 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:56.069+0000 7f9a3ee87640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:39:56.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:39:56.395 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:39:56.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:39:56.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:39:56.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:39:56.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:39:56.873 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3508634204,v1:127.0.0.1:6803/3508634204] [v2:127.0.0.1:6804/3508634204,v1:127.0.0.1:6805/3508634204] exists,up 04642d86-93c8-4ec6-b3d3-29b9389207bb 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/1 2024-09-10T19:39:56.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:39:56.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:39:56.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/1 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be3d1bf6-c61a-4b2c-b932-194b8ac21bc0 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 be3d1bf6-c61a-4b2c-b932-194b8ac21bc0' 2024-09-10T19:39:56.880 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 be3d1bf6-c61a-4b2c-b932-194b8ac21bc0 2024-09-10T19:39:56.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:39:56.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCMoOBmpjmBNRAAoxrncF+nGbcbBMwQb3FTTA== 2024-09-10T19:39:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCMoOBmpjmBNRAAoxrncF+nGbcbBMwQb3FTTA=="}' 2024-09-10T19:39:56.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be3d1bf6-c61a-4b2c-b932-194b8ac21bc0 -i td/osd-backfill-prio/1/new.json 2024-09-10T19:39:57.406 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:39:57.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/1/new.json 2024-09-10T19:39:57.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMoOBmpjmBNRAAoxrncF+nGbcbBMwQb3FTTA== --osd-uuid be3d1bf6-c61a-4b2c-b932-194b8ac21bc0 2024-09-10T19:39:57.454 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:57.453+0000 7f3e7800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:57.492 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:57.490+0000 7f3e7800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:57.493 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:57.491+0000 7f3e7800b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:39:57.493 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:57.491+0000 7f3e7800b640 -1 bdev(0x55aec1db7c00 td/osd-backfill-prio/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:39:57.493 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:39:57.492+0000 7f3e7800b640 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2024-09-10T19:40:04.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2024-09-10T19:40:04.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:40:04.009 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:40:04.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:40:04.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:40:04.546 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:40:04.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:40:04.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:40:04.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:40:04.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:40:04.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:40:04.580 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:04.579+0000 7f85582ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:04.588 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:04.587+0000 7f85582ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:04.589 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:04.588+0000 7f85582ef640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:05.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:40:05.004 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:40:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:05.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:40:05.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:06.182 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:06.180+0000 7f85582ef640 -1 Falling back to public interface 2024-09-10T19:40:06.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:06.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:06.431 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:40:06.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:40:06.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:06.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:40:06.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:07.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:07.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:07.830 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:40:07.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:40:07.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:07.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:40:08.000 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:07.998+0000 7f85582ef640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:40:08.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:09.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:09.319 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:40:09.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:40:09.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:09.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:40:09.750 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1149470806,v1:127.0.0.1:6811/1149470806] [v2:127.0.0.1:6812/1149470806,v1:127.0.0.1:6813/1149470806] exists,up be3d1bf6-c61a-4b2c-b932-194b8ac21bc0 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:40:09.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/2 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:40:09.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:40:09.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:40:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:40:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:40:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/2 2024-09-10T19:40:09.756 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:40:09.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5acb21d4-cefb-4cb7-b31f-91aeca15d83f 2024-09-10T19:40:09.757 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 5acb21d4-cefb-4cb7-b31f-91aeca15d83f 2024-09-10T19:40:09.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5acb21d4-cefb-4cb7-b31f-91aeca15d83f' 2024-09-10T19:40:09.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:40:09.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCZoOBmoMYyLhAA6uVHYFc7v6duWUdaW3QPBA== 2024-09-10T19:40:09.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCZoOBmoMYyLhAA6uVHYFc7v6duWUdaW3QPBA=="}' 2024-09-10T19:40:09.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5acb21d4-cefb-4cb7-b31f-91aeca15d83f -i td/osd-backfill-prio/2/new.json 2024-09-10T19:40:10.269 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:40:10.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/2/new.json 2024-09-10T19:40:10.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZoOBmoMYyLhAA6uVHYFc7v6duWUdaW3QPBA== --osd-uuid 5acb21d4-cefb-4cb7-b31f-91aeca15d83f 2024-09-10T19:40:10.316 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:10.315+0000 7f4566a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:10.317+0000 7f4566a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:10.319 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:10.318+0000 7f4566a31640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:10.320 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:10.318+0000 7f4566a31640 -1 bdev(0x55724bbffc00 td/osd-backfill-prio/2/block) open stat got: (1) Operation not permitted 2024-09-10T19:40:10.320 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:10.318+0000 7f4566a31640 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2024-09-10T19:40:16.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2024-09-10T19:40:16.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:40:16.241 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T19:40:16.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T19:40:16.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:40:16.781 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T19:40:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T19:40:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:40:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:40:16.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:40:16.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:40:16.815 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:16.813+0000 7f403f213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:16.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:16.815+0000 7f403f213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:16.818 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:16.816+0000 7f403f213640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:17.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T19:40:17.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:40:17.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T19:40:17.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:40:17.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:40:17.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:17.183 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:40:17.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:40:17.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:17.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:40:17.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:18.409 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:18.406+0000 7f403f213640 -1 Falling back to public interface 2024-09-10T19:40:18.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:18.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:18.613 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:40:18.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:40:18.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:18.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:40:19.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:19.975 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:19.972+0000 7f403f213640 -1 osd.2 0 log_to_monitors true 2024-09-10T19:40:20.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:20.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:20.064 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:40:20.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:40:20.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:20.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:40:20.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:21.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:21.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:21.629 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:40:21.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:40:21.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:21.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:40:22.027 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2773043723,v1:127.0.0.1:6819/2773043723] [v2:127.0.0.1:6820/2773043723,v1:127.0.0.1:6821/2773043723] exists,up 5acb21d4-cefb-4cb7-b31f-91aeca15d83f 2024-09-10T19:40:22.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/3 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:40:22.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:40:22.033 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:40:22.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/3 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e776d9d1-c6d5-41c7-892c-94287c14ffa3 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 e776d9d1-c6d5-41c7-892c-94287c14ffa3 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e776d9d1-c6d5-41c7-892c-94287c14ffa3' 2024-09-10T19:40:22.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:40:22.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCmoOBmoA8WAxAAxpDDnm0CByybfoOIdMq/Gw== 2024-09-10T19:40:22.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCmoOBmoA8WAxAAxpDDnm0CByybfoOIdMq/Gw=="}' 2024-09-10T19:40:22.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e776d9d1-c6d5-41c7-892c-94287c14ffa3 -i td/osd-backfill-prio/3/new.json 2024-09-10T19:40:22.522 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:40:22.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/3/new.json 2024-09-10T19:40:22.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmoOBmoA8WAxAAxpDDnm0CByybfoOIdMq/Gw== --osd-uuid e776d9d1-c6d5-41c7-892c-94287c14ffa3 2024-09-10T19:40:22.570 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:22.567+0000 7f2ffa013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:22.597 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:22.595+0000 7f2ffa013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:22.598 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:22.596+0000 7f2ffa013640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:22.599 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:22.596+0000 7f2ffa013640 -1 bdev(0x560a3e9b7c00 td/osd-backfill-prio/3/block) open stat got: (1) Operation not permitted 2024-09-10T19:40:22.599 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:22.596+0000 7f2ffa013640 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2024-09-10T19:40:28.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2024-09-10T19:40:28.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:40:28.854 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-10T19:40:28.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T19:40:28.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:40:29.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T19:40:29.396 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-10T19:40:29.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:40:29.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:40:29.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:40:29.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:40:29.430 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:29.428+0000 7f6b22ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:29.446 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:29.445+0000 7f6b22ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:29.447 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:29.446+0000 7f6b22ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:40:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:40:29.854 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:40:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:40:30.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:31.035 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:31.033+0000 7f6b22ab6640 -1 Falling back to public interface 2024-09-10T19:40:31.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:31.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:31.288 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:40:31.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:40:31.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:31.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:40:31.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:32.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:32.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:32.714 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:40:32.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:40:32.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:32.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:40:33.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:33.243 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:33.241+0000 7f6b22ab6640 -1 osd.3 0 log_to_monitors true 2024-09-10T19:40:34.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:34.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:34.113 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:40:34.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:40:34.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:34.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:40:34.506 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3399460140,v1:127.0.0.1:6827/3399460140] [v2:127.0.0.1:6828/3399460140,v1:127.0.0.1:6829/3399460140] exists,up e776d9d1-c6d5-41c7-892c-94287c14ffa3 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:60: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:62: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-prio 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-prio/4 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq ' 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:40:34.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' 2024-09-10T19:40:34.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-prio/4 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3938a32c-dc9e-4676-996b-c1712d6ebd9c 2024-09-10T19:40:34.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 3938a32c-dc9e-4676-996b-c1712d6ebd9c' 2024-09-10T19:40:34.513 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 3938a32c-dc9e-4676-996b-c1712d6ebd9c 2024-09-10T19:40:34.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:40:34.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCyoOBmX964HxAAIorDysLI2azVDeFNaCX3MA== 2024-09-10T19:40:34.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCyoOBmX964HxAAIorDysLI2azVDeFNaCX3MA=="}' 2024-09-10T19:40:34.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3938a32c-dc9e-4676-996b-c1712d6ebd9c -i td/osd-backfill-prio/4/new.json 2024-09-10T19:40:35.023 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:40:35.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-prio/4/new.json 2024-09-10T19:40:35.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyoOBmX964HxAAIorDysLI2azVDeFNaCX3MA== --osd-uuid 3938a32c-dc9e-4676-996b-c1712d6ebd9c 2024-09-10T19:40:35.070 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:35.068+0000 7fedb32a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:35.101 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:35.099+0000 7fedb32a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:35.102 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:35.100+0000 7fedb32a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:35.102 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:35.100+0000 7fedb32a5640 -1 bdev(0x55fa8bf3fc00 td/osd-backfill-prio/4/block) open stat got: (1) Operation not permitted 2024-09-10T19:40:35.103 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:35.100+0000 7fedb32a5640 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2024-09-10T19:40:41.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2024-09-10T19:40:41.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:40:41.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T19:40:41.149 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-10T19:40:41.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:40:41.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T19:40:41.694 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-10T19:40:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=d666df1d-183c-4911-805a-0b592f003f30 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.27247/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:40:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:40:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:40:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:40:41.727 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:41.726+0000 7f52d85d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:41.739 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:41.738+0000 7f52d85d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:41.740 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:41.739+0000 7f52d85d9640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:40:42.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T19:40:42.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:40:42.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T19:40:42.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:42.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:40:42.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:42.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:42.833+0000 7f52d85d9640 -1 Falling back to public interface 2024-09-10T19:40:43.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:43.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:43.588 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:40:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:40:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:43.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:40:44.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:44.835 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:40:44.833+0000 7f52d85d9640 -1 osd.4 0 log_to_monitors true 2024-09-10T19:40:45.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:45.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:45.016 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:40:45.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:40:45.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:45.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:40:45.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:40:46.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:40:46.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:40:46.631 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:40:46.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:40:46.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:40:46.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:40:47.099 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1510260501,v1:127.0.0.1:6835/1510260501] [v2:127.0.0.1:6836/1510260501,v1:127.0.0.1:6837/1510260501] exists,up 3938a32c-dc9e-4676-996b-c1712d6ebd9c 2024-09-10T19:40:47.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:40:47.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:40:47.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:40:47.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: seq 1 10 2024-09-10T19:40:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test1 1 1 2024-09-10T19:40:47.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test1 1 1 2024-09-10T19:40:47.628 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' already exists 2024-09-10T19:40:47.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:40:48.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test1 size 2 2024-09-10T19:40:49.427 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 2 2024-09-10T19:40:49.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:49.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test2 1 1 2024-09-10T19:40:49.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test2 1 1 2024-09-10T19:40:50.159 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' already exists 2024-09-10T19:40:50.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:40:51.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test2 size 2 2024-09-10T19:40:51.949 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 2 size to 2 2024-09-10T19:40:51.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:51.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test3 1 1 2024-09-10T19:40:51.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test3 1 1 2024-09-10T19:40:52.584 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' already exists 2024-09-10T19:40:52.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:40:53.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test3 size 2 2024-09-10T19:40:54.524 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2024-09-10T19:40:54.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:54.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test4 1 1 2024-09-10T19:40:54.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test4 1 1 2024-09-10T19:40:55.277 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test4' already exists 2024-09-10T19:40:55.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:40:56.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test4 size 2 2024-09-10T19:40:57.475 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 4 size to 2 2024-09-10T19:40:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test5 1 1 2024-09-10T19:40:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test5 1 1 2024-09-10T19:40:58.168 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test5' already exists 2024-09-10T19:40:58.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:40:59.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test5 size 2 2024-09-10T19:40:59.795 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 5 size to 2 2024-09-10T19:40:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:40:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test6 1 1 2024-09-10T19:40:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test6 1 1 2024-09-10T19:41:00.570 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test6' already exists 2024-09-10T19:41:00.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:41:01.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test6 size 2 2024-09-10T19:41:02.251 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 6 size to 2 2024-09-10T19:41:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test7 1 1 2024-09-10T19:41:02.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test7 1 1 2024-09-10T19:41:03.124 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test7' already exists 2024-09-10T19:41:03.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:41:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test7 size 2 2024-09-10T19:41:05.033 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 2 2024-09-10T19:41:05.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:05.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test8 1 1 2024-09-10T19:41:05.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test8 1 1 2024-09-10T19:41:05.710 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test8' already exists 2024-09-10T19:41:05.721 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:41:06.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test8 size 2 2024-09-10T19:41:07.623 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 8 size to 2 2024-09-10T19:41:07.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:07.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test9 1 1 2024-09-10T19:41:07.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test9 1 1 2024-09-10T19:41:08.579 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test9' already exists 2024-09-10T19:41:08.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:41:09.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test9 size 2 2024-09-10T19:41:10.511 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 9 size to 2 2024-09-10T19:41:10.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:65: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:10.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:67: TEST_backfill_priority: create_pool test10 1 1 2024-09-10T19:41:10.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test10 1 1 2024-09-10T19:41:11.218 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test10' already exists 2024-09-10T19:41:11.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:41:12.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:68: TEST_backfill_priority: ceph osd pool set test10 size 2 2024-09-10T19:41:13.476 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 10 size to 2 2024-09-10T19:41:13.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:70: TEST_backfill_priority: sleep 5 2024-09-10T19:41:18.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:72: TEST_backfill_priority: wait_for_clean 2024-09-10T19:41:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:41:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:41:18.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:41:18.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:41:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:41:18.493 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:41:18.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:41:18.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:41:18.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:41:18.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:41:18.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:41:18.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:41:18.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:41:18.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:41:18.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:19.055 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:41:19.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836499 2024-09-10T19:41:19.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836499 2024-09-10T19:41:19.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499' 2024-09-10T19:41:19.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:19.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:41:19.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-10T19:41:19.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-10T19:41:19.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976' 2024-09-10T19:41:19.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:19.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:41:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509454 2024-09-10T19:41:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509454 2024-09-10T19:41:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454' 2024-09-10T19:41:19.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:19.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:41:19.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345931 2024-09-10T19:41:19.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345931 2024-09-10T19:41:19.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931' 2024-09-10T19:41:19.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:19.531 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:41:19.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182409 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182409 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836499 1-42949672976 2-64424509454 3-85899345931 4-107374182409' 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836499 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:19.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:41:19.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836499 2024-09-10T19:41:19.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:19.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836499 2024-09-10T19:41:19.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2024-09-10T19:41:19.654 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836499 2024-09-10T19:41:19.654 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:41:20.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836497 -lt 21474836499 2024-09-10T19:41:20.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:41:21.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:41:21.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:41:21.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836499 2024-09-10T19:41:21.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:21.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-10T19:41:21.526 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:21.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:41:21.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-10T19:41:21.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-10T19:41:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-10T19:41:21.528 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672976 2024-09-10T19:41:21.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:41:21.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-10T19:41:21.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:21.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509454 2024-09-10T19:41:21.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:21.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:41:21.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509454 2024-09-10T19:41:21.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:21.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509454 2024-09-10T19:41:21.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-10T19:41:21.963 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509454 2024-09-10T19:41:21.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:41:22.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-10T19:41:22.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:22.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345931 2024-09-10T19:41:22.353 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:22.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:41:22.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345931 2024-09-10T19:41:22.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:22.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345931 2024-09-10T19:41:22.355 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345931 2024-09-10T19:41:22.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2024-09-10T19:41:22.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:41:22.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345932 -lt 85899345931 2024-09-10T19:41:22.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:22.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182409 2024-09-10T19:41:22.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:22.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:41:22.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182409 2024-09-10T19:41:22.749 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:22.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182409 2024-09-10T19:41:22.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-09-10T19:41:22.750 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182409 2024-09-10T19:41:22.750 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:41:23.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182410 -lt 107374182409 2024-09-10T19:41:23.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:41:23.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:41:23.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:41:23.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 10 == 0 2024-09-10T19:41:23.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:41:23.707 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:41:24.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=10 2024-09-10T19:41:24.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:41:24.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:41:24.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:41:24.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 = 10 2024-09-10T19:41:24.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:41:24.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:41:24.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:74: TEST_backfill_priority: ceph pg dump pgs 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:14.116726+0000 0'0 80:25 [2,1] 2 [2,1] 2 0'0 2024-09-10T19:41:10.849629+0000 0'0 2024-09-10T19:41:10.849629+0000 0 0 periodic scrub scheduled @ 2024-09-12T06:59:34.824602+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:10.951517+0000 0'0 80:37 [1,2] 1 [1,2] 1 0'0 2024-09-10T19:41:08.293226+0000 0'0 2024-09-10T19:41:08.293226+0000 0 0 periodic scrub scheduled @ 2024-09-12T07:25:54.052259+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:08.830162+0000 0'0 80:40 [3,4] 3 [3,4] 3 0'0 2024-09-10T19:41:05.425320+0000 0'0 2024-09-10T19:41:05.425320+0000 0 0 periodic scrub scheduled @ 2024-09-11T22:54:51.706500+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:05.929373+0000 0'0 80:56 [1,4] 1 [1,4] 1 0'0 2024-09-10T19:41:02.660463+0000 0'0 2024-09-10T19:41:02.660463+0000 0 0 periodic scrub scheduled @ 2024-09-12T06:07:07.847914+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:04.035396+0000 0'0 80:56 [4,2] 4 [4,2] 4 0'0 2024-09-10T19:41:00.188822+0000 0'0 2024-09-10T19:41:00.188822+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:45:02.095667+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:41:00.675573+0000 0'0 80:67 [3,4] 3 [3,4] 3 0'0 2024-09-10T19:40:57.890833+0000 0'0 2024-09-10T19:40:57.890833+0000 0 0 periodic scrub scheduled @ 2024-09-12T06:31:19.421683+0000 0 0 2024-09-10T19:41:25.021 INFO:tasks.workunit.client.0.smithi130.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:40:58.411361+0000 0'0 80:77 [4,0] 4 [4,0] 4 0'0 2024-09-10T19:40:54.932127+0000 0'0 2024-09-10T19:40:54.932127+0000 0 0 periodic scrub scheduled @ 2024-09-12T02:27:36.335813+0000 0 0 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:40:55.076052+0000 0'0 80:95 [1,2] 1 [1,2] 1 0'0 2024-09-10T19:40:52.405381+0000 0'0 2024-09-10T19:40:52.405381+0000 0 0 periodic scrub scheduled @ 2024-09-12T01:54:33.394843+0000 0 0 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:40:52.938734+0000 0'0 80:87 [3,1] 3 [3,1] 3 0'0 2024-09-10T19:40:49.806373+0000 0'0 2024-09-10T19:40:49.806373+0000 0 0 periodic scrub scheduled @ 2024-09-12T02:00:24.042541+0000 0 0 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-10T19:40:49.714580+0000 0'0 80:104 [1,0] 1 [1,0] 1 0'0 2024-09-10T19:40:47.465055+0000 0'0 2024-09-10T19:40:47.465055+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:34:15.964530+0000 0 0 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:41:25.022 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:41:25.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:78: TEST_backfill_priority: local PG1 2024-09-10T19:41:25.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:79: TEST_backfill_priority: local POOLNUM1 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:80: TEST_backfill_priority: local pool1 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:81: TEST_backfill_priority: local chk_osd1_1 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:82: TEST_backfill_priority: local chk_osd1_2 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:84: TEST_backfill_priority: local PG2 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:85: TEST_backfill_priority: local POOLNUM2 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:86: TEST_backfill_priority: local pool2 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:87: TEST_backfill_priority: local chk_osd2 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:89: TEST_backfill_priority: local PG3 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:90: TEST_backfill_priority: local POOLNUM3 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:91: TEST_backfill_priority: local pool3 2024-09-10T19:41:25.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: seq 1 10 2024-09-10T19:41:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 1.0 --format=json 2024-09-10T19:41:25.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:25.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:25.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2024-09-10T19:41:25.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:25.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2024-09-10T19:41:25.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z '' ']' 2024-09-10T19:41:25.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:100: TEST_backfill_priority: PG1=1.0 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:101: TEST_backfill_priority: POOLNUM1=1 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:102: TEST_backfill_priority: pool1=test1 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:103: TEST_backfill_priority: chk_osd1_1=1 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:104: TEST_backfill_priority: chk_osd1_2=0 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 2.0 --format=json 2024-09-10T19:41:25.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:25.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:25.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2024-09-10T19:41:25.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=1 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 3.0 --format=json 2024-09-10T19:41:25.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:26.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:26.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2024-09-10T19:41:26.297 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:107: TEST_backfill_priority: PG2=3.0 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:108: TEST_backfill_priority: POOLNUM2=3 2024-09-10T19:41:26.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:109: TEST_backfill_priority: pool2=test3 2024-09-10T19:41:26.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:110: TEST_backfill_priority: chk_osd2=2 2024-09-10T19:41:26.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:26.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 4.0 --format=json 2024-09-10T19:41:26.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:26.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:26.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2024-09-10T19:41:26.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=0 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 5.0 --format=json 2024-09-10T19:41:26.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:27.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=3 2024-09-10T19:41:27.116 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 6.0 --format=json 2024-09-10T19:41:27.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:27.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=4 2024-09-10T19:41:27.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=2 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:93: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: ceph pg map 7.0 --format=json 2024-09-10T19:41:27.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:95: TEST_backfill_priority: jq '.acting[]' 2024-09-10T19:41:27.981 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:96: TEST_backfill_priority: local test_osd1=1 2024-09-10T19:41:27.982 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:97: TEST_backfill_priority: local test_osd2=4 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:98: TEST_backfill_priority: '[' -z 1.0 ']' 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:105: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:111: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:113: TEST_backfill_priority: PG3=7.0 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:114: TEST_backfill_priority: POOLNUM3=7 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:115: TEST_backfill_priority: pool3=test7 2024-09-10T19:41:27.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:116: TEST_backfill_priority: break 2024-09-10T19:41:27.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:119: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2024-09-10T19:41:27.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:121: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2024-09-10T19:41:27.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: seq 1 10 2024-09-10T19:41:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-09-10T19:41:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:27.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-09-10T19:41:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test2 2024-09-10T19:41:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test2 2024-09-10T19:41:27.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-10T19:41:28.489 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test2' does not exist 2024-09-10T19:41:28.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:28.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-09-10T19:41:28.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:28.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-09-10T19:41:28.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test4 2024-09-10T19:41:28.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test4 2024-09-10T19:41:28.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-09-10T19:41:28.990 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test4' does not exist 2024-09-10T19:41:29.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-09-10T19:41:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test5 2024-09-10T19:41:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test5 2024-09-10T19:41:29.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-09-10T19:41:29.579 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test5' does not exist 2024-09-10T19:41:29.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:29.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-09-10T19:41:29.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test6 2024-09-10T19:41:29.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test6 2024-09-10T19:41:29.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-09-10T19:41:30.166 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test6' does not exist 2024-09-10T19:41:30.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:30.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-09-10T19:41:30.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:30.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-09-10T19:41:30.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test8 2024-09-10T19:41:30.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test8 2024-09-10T19:41:30.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-09-10T19:41:30.817 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test8' does not exist 2024-09-10T19:41:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-09-10T19:41:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test9 2024-09-10T19:41:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test9 2024-09-10T19:41:30.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-09-10T19:41:31.497 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test9' does not exist 2024-09-10T19:41:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:127: TEST_backfill_priority: for p in $(seq 1 $pools) 2024-09-10T19:41:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:129: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-09-10T19:41:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:131: TEST_backfill_priority: delete_pool test10 2024-09-10T19:41:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test10 2024-09-10T19:41:31.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-09-10T19:41:32.244 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test10' does not exist 2024-09-10T19:41:32.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:135: TEST_backfill_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-09-10T19:41:33.064 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 1 2024-09-10T19:41:33.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:136: TEST_backfill_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-09-10T19:41:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 1 2024-09-10T19:41:34.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:137: TEST_backfill_priority: wait_for_clean 2024-09-10T19:41:34.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:41:34.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:41:34.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:41:34.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:41:34.024 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:41:34.024 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:41:34.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:41:34.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:41:34.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:41:34.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:41:34.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:41:34.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:41:34.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:41:34.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:41:34.111 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:34.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:41:34.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836504 2024-09-10T19:41:34.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836504 2024-09-10T19:41:34.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504' 2024-09-10T19:41:34.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:34.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:41:34.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672982 2024-09-10T19:41:34.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672982 2024-09-10T19:41:34.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672982' 2024-09-10T19:41:34.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:34.833 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:41:34.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509459 2024-09-10T19:41:34.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509459 2024-09-10T19:41:34.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459' 2024-09-10T19:41:34.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:34.989 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:41:35.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345937 2024-09-10T19:41:35.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345937 2024-09-10T19:41:35.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459 3-85899345937' 2024-09-10T19:41:35.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:41:35.109 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:41:35.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182414 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182414 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836504 1-42949672982 2-64424509459 3-85899345937 4-107374182414' 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836504 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:35.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:41:35.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836504 2024-09-10T19:41:35.230 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:35.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836504 2024-09-10T19:41:35.231 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836504 2024-09-10T19:41:35.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-10T19:41:35.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:41:35.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-10T19:41:35.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:41:36.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:41:36.659 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:41:37.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836504 -lt 21474836504 2024-09-10T19:41:37.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:37.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672982 2024-09-10T19:41:37.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:41:37.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672982 2024-09-10T19:41:37.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672982 2024-09-10T19:41:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-10T19:41:37.060 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672982 2024-09-10T19:41:37.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:41:37.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-10T19:41:37.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:37.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509459 2024-09-10T19:41:37.513 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:37.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:41:37.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509459 2024-09-10T19:41:37.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:37.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509459 2024-09-10T19:41:37.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-10T19:41:37.515 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509459 2024-09-10T19:41:37.515 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:41:37.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-10T19:41:37.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:37.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345937 2024-09-10T19:41:37.910 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:37.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:41:37.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345937 2024-09-10T19:41:37.911 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:37.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345937 2024-09-10T19:41:37.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-10T19:41:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345937 2024-09-10T19:41:37.913 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:41:38.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-10T19:41:38.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:41:38.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182414 2024-09-10T19:41:38.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:41:38.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:41:38.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182414 2024-09-10T19:41:38.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:41:38.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182414 2024-09-10T19:41:38.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-09-10T19:41:38.359 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182414 2024-09-10T19:41:38.359 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:41:38.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182415 -lt 107374182414 2024-09-10T19:41:38.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:41:38.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:41:38.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:41:39.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-10T19:41:39.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:41:39.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:41:39.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:41:39.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:41:39.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:41:39.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:41:39.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:41:39.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T19:41:39.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:41:39.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:41:39.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:41:40.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-10T19:41:40.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:41:40.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:41:40.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:139: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records in 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:10+0 records out 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0288695 s, 363 MB/s 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:140: TEST_backfill_priority: p=1 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2024-09-10T19:41:40.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2024-09-10T19:41:40.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:40.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2024-09-10T19:41:40.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:40.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2024-09-10T19:41:41.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:41.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2024-09-10T19:41:42.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:42.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2024-09-10T19:41:42.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:42.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2024-09-10T19:41:43.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:43.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2024-09-10T19:41:44.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:44.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2024-09-10T19:41:44.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:44.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2024-09-10T19:41:45.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:45.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2024-09-10T19:41:46.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:46.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2024-09-10T19:41:46.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:46.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2024-09-10T19:41:47.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:47.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2024-09-10T19:41:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:48.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2024-09-10T19:41:49.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:49.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2024-09-10T19:41:49.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:49.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2024-09-10T19:41:50.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:50.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2024-09-10T19:41:50.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:50.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2024-09-10T19:41:51.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2024-09-10T19:41:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:52.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2024-09-10T19:41:52.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:52.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2024-09-10T19:41:53.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:53.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2024-09-10T19:41:54.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:54.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2024-09-10T19:41:55.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:55.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2024-09-10T19:41:55.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:55.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2024-09-10T19:41:56.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:56.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2024-09-10T19:41:57.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:57.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2024-09-10T19:41:57.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:57.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2024-09-10T19:41:58.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:58.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2024-09-10T19:41:58.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:58.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2024-09-10T19:41:59.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:41:59.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2024-09-10T19:42:00.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:00.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2024-09-10T19:42:00.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:00.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2024-09-10T19:42:01.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:01.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2024-09-10T19:42:02.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:02.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2024-09-10T19:42:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:02.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2024-09-10T19:42:03.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:03.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2024-09-10T19:42:04.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:04.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2024-09-10T19:42:04.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:04.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2024-09-10T19:42:05.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:05.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2024-09-10T19:42:06.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:06.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2024-09-10T19:42:06.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:06.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2024-09-10T19:42:07.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:07.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2024-09-10T19:42:08.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:08.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2024-09-10T19:42:08.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:08.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2024-09-10T19:42:09.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:09.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2024-09-10T19:42:10.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:10.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2024-09-10T19:42:10.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:10.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2024-09-10T19:42:11.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:11.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2024-09-10T19:42:12.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:12.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2024-09-10T19:42:12.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:12.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2024-09-10T19:42:13.429 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 1 + 1 2024-09-10T19:42:13.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=2 2024-09-10T19:42:13.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2024-09-10T19:42:13.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2024-09-10T19:42:13.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:13.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2024-09-10T19:42:13.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:13.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2024-09-10T19:42:14.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:14.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2024-09-10T19:42:14.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:14.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2024-09-10T19:42:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:15.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2024-09-10T19:42:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:15.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2024-09-10T19:42:15.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:15.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2024-09-10T19:42:16.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:16.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2024-09-10T19:42:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:16.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2024-09-10T19:42:16.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:16.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2024-09-10T19:42:17.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:17.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2024-09-10T19:42:17.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:17.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2024-09-10T19:42:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:17.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2024-09-10T19:42:18.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:18.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2024-09-10T19:42:18.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:18.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2024-09-10T19:42:19.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:19.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2024-09-10T19:42:19.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:19.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2024-09-10T19:42:20.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:20.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2024-09-10T19:42:20.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:20.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2024-09-10T19:42:20.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:20.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2024-09-10T19:42:21.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:21.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2024-09-10T19:42:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:21.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2024-09-10T19:42:21.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:21.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2024-09-10T19:42:22.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:22.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2024-09-10T19:42:22.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:22.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2024-09-10T19:42:23.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:23.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2024-09-10T19:42:23.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:23.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2024-09-10T19:42:23.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:23.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2024-09-10T19:42:24.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:24.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2024-09-10T19:42:24.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:24.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2024-09-10T19:42:25.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:25.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2024-09-10T19:42:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:25.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2024-09-10T19:42:25.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:25.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2024-09-10T19:42:26.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:26.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2024-09-10T19:42:26.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:26.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2024-09-10T19:42:26.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:26.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2024-09-10T19:42:27.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:27.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2024-09-10T19:42:27.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:27.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2024-09-10T19:42:27.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:27.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2024-09-10T19:42:28.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:28.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2024-09-10T19:42:28.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:28.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2024-09-10T19:42:29.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:29.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2024-09-10T19:42:29.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:29.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2024-09-10T19:42:29.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:29.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2024-09-10T19:42:30.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:30.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2024-09-10T19:42:30.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:30.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2024-09-10T19:42:30.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:30.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2024-09-10T19:42:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2024-09-10T19:42:31.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:31.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2024-09-10T19:42:31.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:31.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2024-09-10T19:42:32.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 2 + 1 2024-09-10T19:42:32.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=3 2024-09-10T19:42:32.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:141: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2024-09-10T19:42:32.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: seq 1 50 2024-09-10T19:42:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:32.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2024-09-10T19:42:32.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:32.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2024-09-10T19:42:33.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:33.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2024-09-10T19:42:33.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:33.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2024-09-10T19:42:33.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:33.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2024-09-10T19:42:34.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:34.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2024-09-10T19:42:34.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:34.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2024-09-10T19:42:34.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:34.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2024-09-10T19:42:35.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:35.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2024-09-10T19:42:35.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:35.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2024-09-10T19:42:35.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:35.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2024-09-10T19:42:36.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:36.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2024-09-10T19:42:36.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:36.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2024-09-10T19:42:36.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:36.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2024-09-10T19:42:37.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:37.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2024-09-10T19:42:37.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:37.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2024-09-10T19:42:38.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:38.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2024-09-10T19:42:38.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:38.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2024-09-10T19:42:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:38.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2024-09-10T19:42:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:39.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2024-09-10T19:42:39.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:39.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2024-09-10T19:42:39.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:39.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2024-09-10T19:42:40.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:40.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2024-09-10T19:42:40.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:40.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2024-09-10T19:42:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2024-09-10T19:42:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2024-09-10T19:42:41.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:41.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2024-09-10T19:42:42.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:42.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2024-09-10T19:42:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:42.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2024-09-10T19:42:42.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:42.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2024-09-10T19:42:43.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:43.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2024-09-10T19:42:43.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:43.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2024-09-10T19:42:43.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:43.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2024-09-10T19:42:44.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:44.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2024-09-10T19:42:44.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:44.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2024-09-10T19:42:44.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:44.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2024-09-10T19:42:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:45.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2024-09-10T19:42:45.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:45.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2024-09-10T19:42:45.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:45.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2024-09-10T19:42:46.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:46.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2024-09-10T19:42:46.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:46.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2024-09-10T19:42:47.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:47.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2024-09-10T19:42:47.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:47.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2024-09-10T19:42:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2024-09-10T19:42:48.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:48.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2024-09-10T19:42:48.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:48.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2024-09-10T19:42:48.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:48.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2024-09-10T19:42:49.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:49.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2024-09-10T19:42:49.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:49.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2024-09-10T19:42:50.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:143: TEST_backfill_priority: for i in $(seq 1 $objects) 2024-09-10T19:42:50.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:145: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2024-09-10T19:42:50.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: expr 3 + 1 2024-09-10T19:42:50.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:147: TEST_backfill_priority: p=4 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: get_not_primary test1 obj1-p1 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test1 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1-p1 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test1 obj1-p1 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test1 2024-09-10T19:42:50.469 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1-p1 2024-09-10T19:42:50.470 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-10T19:42:50.470 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T19:42:50.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T19:42:50.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-10T19:42:50.895 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T19:42:51.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:150: TEST_backfill_priority: local otherosd=0 2024-09-10T19:42:51.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph pg dump pgs 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 48 0 0 0 0 497025024 0 0 42 100 42 active+clean 2024-09-10T19:41:34.564016+0000 92'142 92:220 [1] 1 [1] 1 0'0 2024-09-10T19:41:02.660463+0000 0'0 2024-09-10T19:41:02.660463+0000 0 0 periodic scrub scheduled @ 2024-09-12T00:39:12.513043+0000 0 0 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:41:34.642575+0000 92'150 92:265 [1] 1 [1] 1 0'0 2024-09-10T19:40:52.405381+0000 0'0 2024-09-10T19:40:52.405381+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:29:23.769492+0000 0 0 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:40:49.714580+0000 92'150 92:273 [1,0] 1 [1,0] 1 0'0 2024-09-10T19:40:47.465055+0000 0'0 2024-09-10T19:40:47.465055+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:34:15.964530+0000 0 0 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:42:51.705 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:42:51.706 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:42:51.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:153: TEST_backfill_priority: ERRORS=0 2024-09-10T19:42:51.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:155: TEST_backfill_priority: ceph osd set nobackfill 2024-09-10T19:42:52.359 INFO:tasks.workunit.client.0.smithi130.stderr:nobackfill is set 2024-09-10T19:42:52.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:156: TEST_backfill_priority: ceph osd set noout 2024-09-10T19:42:53.157 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-10T19:42:53.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph osd pool set test7 size 2 2024-09-10T19:42:53.886 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 7 size to 2 2024-09-10T19:42:53.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:161: TEST_backfill_priority: sleep 2 2024-09-10T19:42:55.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:42:55.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:42:55.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:42:55.901 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:42:55.901 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:42:55.902 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:42:55.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:42:55.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:42:55.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:163: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:42:55.989 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:42:55.989 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:42:55.989 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:42:55.989 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:42:55.990 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:42:55.991 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:42:55.991 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:42:55.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: seq 1 10 2024-09-10T19:42:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2024-09-10T19:42:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2024-09-10T19:42:55.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2024-09-10T19:42:56.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:171: TEST_backfill_priority: '[' 1 = 10 ']' 2024-09-10T19:42:56.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:175: TEST_backfill_priority: sleep 2 2024-09-10T19:42:58.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2024-09-10T19:42:58.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2024-09-10T19:42:58.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2024-09-10T19:42:58.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:171: TEST_backfill_priority: '[' 2 = 10 ']' 2024-09-10T19:42:58.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:175: TEST_backfill_priority: sleep 2 2024-09-10T19:43:00.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:166: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2024-09-10T19:43:00.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: ceph pg force-backfill 7.0 2024-09-10T19:43:00.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:168: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2024-09-10T19:43:01.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:169: TEST_backfill_priority: break 2024-09-10T19:43:01.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:177: TEST_backfill_priority: flush_pg_stats 2024-09-10T19:43:01.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:01.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:01.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:01.603 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:01.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836523 2024-09-10T19:43:01.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836523 2024-09-10T19:43:01.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523' 2024-09-10T19:43:01.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:01.722 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673001 2024-09-10T19:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673001 2024-09-10T19:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949673001' 2024-09-10T19:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:01.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:01.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509479 2024-09-10T19:43:01.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509479 2024-09-10T19:43:01.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509479' 2024-09-10T19:43:01.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:01.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:02.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345956 2024-09-10T19:43:02.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345956 2024-09-10T19:43:02.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509479 3-85899345956' 2024-09-10T19:43:02.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:02.080 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:02.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182434 2024-09-10T19:43:02.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182434 2024-09-10T19:43:02.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836523 1-42949673001 2-64424509479 3-85899345956 4-107374182434' 2024-09-10T19:43:02.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:02.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836523 2024-09-10T19:43:02.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:02.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:02.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836523 2024-09-10T19:43:02.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:02.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836523 2024-09-10T19:43:02.201 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836523 2024-09-10T19:43:02.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836523' 2024-09-10T19:43:02.201 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:02.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836523 2024-09-10T19:43:02.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:02.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673001 2024-09-10T19:43:02.588 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:02.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:02.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673001 2024-09-10T19:43:02.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:02.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673001 2024-09-10T19:43:02.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-10T19:43:02.590 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673001 2024-09-10T19:43:02.591 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:03.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673001 -lt 42949673001 2024-09-10T19:43:03.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:03.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509479 2024-09-10T19:43:03.014 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:03.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:03.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509479 2024-09-10T19:43:03.015 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:03.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509479 2024-09-10T19:43:03.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-10T19:43:03.016 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509479 2024-09-10T19:43:03.016 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:03.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-10T19:43:03.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:03.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345956 2024-09-10T19:43:03.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:03.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:03.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345956 2024-09-10T19:43:03.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:03.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345956 2024-09-10T19:43:03.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2024-09-10T19:43:03.459 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345956 2024-09-10T19:43:03.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:03.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345956 -lt 85899345956 2024-09-10T19:43:03.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:03.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182434 2024-09-10T19:43:03.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:03.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:03.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182434 2024-09-10T19:43:03.950 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:03.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182434 2024-09-10T19:43:03.951 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182434 2024-09-10T19:43:03.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182434' 2024-09-10T19:43:03.952 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:04.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182434 -lt 107374182434 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:04.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:178: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:04.462 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:04.462 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:04.462 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:04.463 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:04.464 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:04.464 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:04.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph osd out osd.0 2024-09-10T19:43:05.072 INFO:tasks.workunit.client.0.smithi130.stderr:osd.0 is already out. 2024-09-10T19:43:05.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:181: TEST_backfill_priority: sleep 2 2024-09-10T19:43:07.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:182: TEST_backfill_priority: flush_pg_stats 2024-09-10T19:43:07.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:07.085 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:07.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:07.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836527 2024-09-10T19:43:07.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836527 2024-09-10T19:43:07.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527' 2024-09-10T19:43:07.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:07.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:07.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673004 2024-09-10T19:43:07.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673004 2024-09-10T19:43:07.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004' 2024-09-10T19:43:07.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:07.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:07.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509482 2024-09-10T19:43:07.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509482 2024-09-10T19:43:07.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-64424509482' 2024-09-10T19:43:07.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:07.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:08.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345959 2024-09-10T19:43:08.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345959 2024-09-10T19:43:08.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-64424509482 3-85899345959' 2024-09-10T19:43:08.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:08.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182437 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182437 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836527 1-42949673004 2-64424509482 3-85899345959 4-107374182437' 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836527 2024-09-10T19:43:08.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:08.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:08.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836527 2024-09-10T19:43:08.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836527 2024-09-10T19:43:08.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-09-10T19:43:08.125 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836527 2024-09-10T19:43:08.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:08.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 0 -lt 21474836527 2024-09-10T19:43:08.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:43:09.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:43:09.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:09.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836527 -lt 21474836527 2024-09-10T19:43:09.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:09.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673004 2024-09-10T19:43:09.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:09.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673004 2024-09-10T19:43:09.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:09.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673004 2024-09-10T19:43:09.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-09-10T19:43:09.975 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673004 2024-09-10T19:43:09.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:10.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673004 -lt 42949673004 2024-09-10T19:43:10.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:10.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509482 2024-09-10T19:43:10.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:10.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:10.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509482 2024-09-10T19:43:10.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:10.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509482 2024-09-10T19:43:10.410 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509482 2024-09-10T19:43:10.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-09-10T19:43:10.410 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:10.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509482 -lt 64424509482 2024-09-10T19:43:10.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:10.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345959 2024-09-10T19:43:10.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:10.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:10.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345959 2024-09-10T19:43:10.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:10.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345959 2024-09-10T19:43:10.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2024-09-10T19:43:10.807 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345959 2024-09-10T19:43:10.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:11.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345960 -lt 85899345959 2024-09-10T19:43:11.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:11.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182437 2024-09-10T19:43:11.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:11.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:11.236 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182437 2024-09-10T19:43:11.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:11.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182437 2024-09-10T19:43:11.237 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182437 2024-09-10T19:43:11.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182437' 2024-09-10T19:43:11.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:11.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182437 -lt 107374182437 2024-09-10T19:43:11.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:11.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:11.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:11.661 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:11.662 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:11.662 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:11.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:11.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:11.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:11.740 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:11.740 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:11.740 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:11.740 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:11.741 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:11.742 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:11.743 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:11.744 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:11.745 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:11.745 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:11.745 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:11.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:184: TEST_backfill_priority: ceph pg dump pgs 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 50 0 0 524288000 0 0 50 100 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2024-09-10T19:43:01.062616+0000 92'150 102:254 [1,4] 1 [1] 1 0'0 2024-09-10T19:41:02.660463+0000 0'0 2024-09-10T19:41:02.660463+0000 0 0 no scrub is scheduled 0 0 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:41:34.642575+0000 92'150 102:286 [1] 1 [1] 1 0'0 2024-09-10T19:40:52.405381+0000 0'0 2024-09-10T19:40:52.405381+0000 0 0 periodic scrub scheduled @ 2024-09-12T03:29:23.769492+0000 0 0 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 50 0 524288000 0 0 50 100 50 active+remapped+backfill_wait 2024-09-10T19:43:06.570926+0000 92'150 102:296 [1,2] 1 [1,0] 1 0'0 2024-09-10T19:40:47.465055+0000 0'0 2024-09-10T19:40:47.465055+0000 0 0 no scrub is scheduled 0 0 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:43:12.115 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:43:12.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph osd pool set test3 size 2 2024-09-10T19:43:12.774 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 3 size to 2 2024-09-10T19:43:12.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:187: TEST_backfill_priority: sleep 2 2024-09-10T19:43:14.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:188: TEST_backfill_priority: flush_pg_stats 2024-09-10T19:43:14.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:14.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:15.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:15.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:15.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836530 2024-09-10T19:43:15.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836530 2024-09-10T19:43:15.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530' 2024-09-10T19:43:15.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:15.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:15.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673008 2024-09-10T19:43:15.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673008 2024-09-10T19:43:15.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008' 2024-09-10T19:43:15.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:15.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:15.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509485 2024-09-10T19:43:15.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509485 2024-09-10T19:43:15.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509485' 2024-09-10T19:43:15.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:15.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:15.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345963 2024-09-10T19:43:15.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345963 2024-09-10T19:43:15.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509485 3-85899345963' 2024-09-10T19:43:15.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:15.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:15.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182440 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182440 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673008 2-64424509485 3-85899345963 4-107374182440' 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836530 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:15.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:15.871 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836530 2024-09-10T19:43:15.871 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:15.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836530 2024-09-10T19:43:15.872 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836530 2024-09-10T19:43:15.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-10T19:43:15.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:16.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836528 -lt 21474836530 2024-09-10T19:43:16.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:43:17.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:43:17.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:17.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-10T19:43:17.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:17.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673008 2024-09-10T19:43:17.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:17.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:17.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673008 2024-09-10T19:43:17.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:17.694 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673008 2024-09-10T19:43:17.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673008 2024-09-10T19:43:17.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-10T19:43:17.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-10T19:43:18.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509485 2024-09-10T19:43:18.089 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:18.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:18.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509485 2024-09-10T19:43:18.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:18.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509485 2024-09-10T19:43:18.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509485' 2024-09-10T19:43:18.092 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509485 2024-09-10T19:43:18.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:18.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509486 -lt 64424509485 2024-09-10T19:43:18.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:18.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345963 2024-09-10T19:43:18.519 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:18.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:18.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345963 2024-09-10T19:43:18.520 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:18.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345963 2024-09-10T19:43:18.521 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345963 2024-09-10T19:43:18.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345963' 2024-09-10T19:43:18.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:18.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345963 -lt 85899345963 2024-09-10T19:43:18.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:18.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182440 2024-09-10T19:43:18.935 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:18.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:18.936 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182440 2024-09-10T19:43:18.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:18.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182440 2024-09-10T19:43:18.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182440' 2024-09-10T19:43:18.938 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182440 2024-09-10T19:43:18.938 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:19.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182441 -lt 107374182440 2024-09-10T19:43:19.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:19.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:19.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:19.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:189: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:19.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:19.484 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:19.485 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:19.486 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:19.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:191: TEST_backfill_priority: ceph pg dump pgs 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 50 0 0 524288000 0 0 50 100 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2024-09-10T19:43:01.062616+0000 92'150 105:259 [1,4] 1 [1] 1 0'0 2024-09-10T19:41:02.660463+0000 0'0 2024-09-10T19:41:02.660463+0000 0 0 no scrub is scheduled 0 0 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 100 50 remapped+peering 2024-09-10T19:43:15.106332+0000 92'150 104:288 [1,2] 1 [1] 1 0'0 2024-09-10T19:40:52.405381+0000 0'0 2024-09-10T19:40:52.405381+0000 0 0 no scrub is scheduled 0 0 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 50 0 524288000 0 0 50 100 50 active+remapped+backfill_wait 2024-09-10T19:43:06.570926+0000 92'150 105:301 [1,2] 1 [1,0] 1 0'0 2024-09-10T19:40:47.465055+0000 0'0 2024-09-10T19:40:47.465055+0000 0 0 no scrub is scheduled 0 0 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:43:19.865 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:43:19.866 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:43:19.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:19.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-09-10T19:43:19.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:193: TEST_backfill_priority: PRIO=110 2024-09-10T19:43:19.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:194: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2024-09-10T19:43:19.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:19.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2024-09-10T19:43:19.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: eval 'ITEM="7.0"' 2024-09-10T19:43:19.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:201: TEST_backfill_priority: ITEM=7.0 2024-09-10T19:43:19.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:202: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-10T19:43:19.882 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:19.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-10T19:43:19.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:207: TEST_backfill_priority: PRIO=254 2024-09-10T19:43:19.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:208: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2024-09-10T19:43:19.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: seq 1 10 2024-09-10T19:43:19.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2024-09-10T19:43:19.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: ceph pg force-backfill 3.0 2024-09-10T19:43:19.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2024-09-10T19:43:20.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:221: TEST_backfill_priority: '[' 1 = 10 ']' 2024-09-10T19:43:20.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:225: TEST_backfill_priority: sleep 2 2024-09-10T19:43:22.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:216: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2024-09-10T19:43:22.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: ceph pg force-backfill 3.0 2024-09-10T19:43:22.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:218: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2024-09-10T19:43:22.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:219: TEST_backfill_priority: break 2024-09-10T19:43:22.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:227: TEST_backfill_priority: sleep 2 2024-09-10T19:43:24.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:24.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:24.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:24.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:228: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:24.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:229: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:24.768 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 254, 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:24.769 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:24.770 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:24.771 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:230: TEST_backfill_priority: PRIO=254 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:231: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:236: TEST_backfill_priority: flush_pg_stats 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:24.772 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:25.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:25.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:25.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836534 2024-09-10T19:43:25.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836534 2024-09-10T19:43:25.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534' 2024-09-10T19:43:25.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:25.307 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:25.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673012 2024-09-10T19:43:25.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673012 2024-09-10T19:43:25.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-42949673012' 2024-09-10T19:43:25.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:25.427 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:25.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509489 2024-09-10T19:43:25.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509489 2024-09-10T19:43:25.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509489' 2024-09-10T19:43:25.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:25.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:25.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345967 2024-09-10T19:43:25.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345967 2024-09-10T19:43:25.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509489 3-85899345967' 2024-09-10T19:43:25.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:25.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182444 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182444 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836534 1-42949673012 2-64424509489 3-85899345967 4-107374182444' 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836534 2024-09-10T19:43:25.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:25.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:25.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836534 2024-09-10T19:43:25.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:25.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836534 2024-09-10T19:43:25.787 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836534 2024-09-10T19:43:25.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-10T19:43:25.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:26.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836532 -lt 21474836534 2024-09-10T19:43:26.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:43:27.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:43:27.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:27.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-10T19:43:27.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:27.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673012 2024-09-10T19:43:27.626 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:27.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:27.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673012 2024-09-10T19:43:27.627 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:27.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673012 2024-09-10T19:43:27.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673012' 2024-09-10T19:43:27.628 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673012 2024-09-10T19:43:27.629 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673012 -lt 42949673012 2024-09-10T19:43:28.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:28.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509489 2024-09-10T19:43:28.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:28.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:28.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509489 2024-09-10T19:43:28.062 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:28.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509489 2024-09-10T19:43:28.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509489' 2024-09-10T19:43:28.064 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509489 2024-09-10T19:43:28.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:28.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509490 -lt 64424509489 2024-09-10T19:43:28.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:28.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345967 2024-09-10T19:43:28.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:28.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:28.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345967 2024-09-10T19:43:28.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:28.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345967 2024-09-10T19:43:28.497 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345967 2024-09-10T19:43:28.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345967' 2024-09-10T19:43:28.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:28.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345967 -lt 85899345967 2024-09-10T19:43:28.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:28.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182444 2024-09-10T19:43:28.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:28.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:28.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182444 2024-09-10T19:43:28.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:28.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182444 2024-09-10T19:43:28.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182444' 2024-09-10T19:43:28.888 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182444 2024-09-10T19:43:28.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:29.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182445 -lt 107374182444 2024-09-10T19:43:29.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:239: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2024-09-10T19:43:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2024-09-10T19:43:29.688 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:43:29.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:240: TEST_backfill_priority: sleep 2 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:31.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:31.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:31.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:241: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:31.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:242: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:31.797 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:31.798 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:31.799 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:31.800 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:243: TEST_backfill_priority: PRIO=151 2024-09-10T19:43:31.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:244: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2024-09-10T19:43:31.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:31.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2024-09-10T19:43:31.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: eval 'ITEM="3.0"' 2024-09-10T19:43:31.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:250: TEST_backfill_priority: ITEM=3.0 2024-09-10T19:43:31.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:251: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2024-09-10T19:43:31.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:31.805 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-10T19:43:31.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:256: TEST_backfill_priority: PRIO=254 2024-09-10T19:43:31.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:257: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2024-09-10T19:43:31.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:264: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2024-09-10T19:43:32.184 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2024-09-10T19:43:32.184 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:43:32.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:265: TEST_backfill_priority: sleep 5 2024-09-10T19:43:37.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:37.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:37.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:37.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:266: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:37.287 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:37.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:37.289 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:37.290 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:37.290 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:37.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:269: TEST_backfill_priority: flush_pg_stats 2024-09-10T19:43:37.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:37.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:37.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:37.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:37.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:37.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836539 2024-09-10T19:43:37.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836539 2024-09-10T19:43:37.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539' 2024-09-10T19:43:37.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:37.837 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673016 2024-09-10T19:43:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673016 2024-09-10T19:43:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-42949673016' 2024-09-10T19:43:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:37.958 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:38.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509494 2024-09-10T19:43:38.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509494 2024-09-10T19:43:38.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-42949673016 2-64424509494' 2024-09-10T19:43:38.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:38.077 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:38.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345971 2024-09-10T19:43:38.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345971 2024-09-10T19:43:38.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-42949673016 2-64424509494 3-85899345971' 2024-09-10T19:43:38.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:38.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182449 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182449 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836539 1-42949673016 2-64424509494 3-85899345971 4-107374182449' 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836539 2024-09-10T19:43:38.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:38.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:38.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836539 2024-09-10T19:43:38.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:38.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836539 2024-09-10T19:43:38.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836539' 2024-09-10T19:43:38.317 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836539 2024-09-10T19:43:38.317 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:38.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836539 -lt 21474836539 2024-09-10T19:43:38.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:38.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673016 2024-09-10T19:43:38.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:38.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:38.710 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673016 2024-09-10T19:43:38.711 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:38.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673016 2024-09-10T19:43:38.711 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673016 2024-09-10T19:43:38.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673016' 2024-09-10T19:43:38.712 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:39.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673016 -lt 42949673016 2024-09-10T19:43:39.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:39.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509494 2024-09-10T19:43:39.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:39.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:39.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509494 2024-09-10T19:43:39.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:39.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509494 2024-09-10T19:43:39.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509494' 2024-09-10T19:43:39.141 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509494 2024-09-10T19:43:39.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:39.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509494 -lt 64424509494 2024-09-10T19:43:39.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:39.572 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345971 2024-09-10T19:43:39.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:39.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:39.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345971 2024-09-10T19:43:39.573 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:39.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345971 2024-09-10T19:43:39.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345971' 2024-09-10T19:43:39.575 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345971 2024-09-10T19:43:39.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:39.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345971 -lt 85899345971 2024-09-10T19:43:39.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:39.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182449 2024-09-10T19:43:39.990 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:39.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:39.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182449 2024-09-10T19:43:39.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:39.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182449 2024-09-10T19:43:39.992 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182449 2024-09-10T19:43:39.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182449' 2024-09-10T19:43:39.992 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:40.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182449 -lt 107374182449 2024-09-10T19:43:40.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph pg force-backfill 7.0 2024-09-10T19:43:40.790 INFO:tasks.workunit.client.0.smithi130.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2024-09-10T19:43:40.790 INFO:tasks.workunit.client.0.smithi130.stderr: 2024-09-10T19:43:40.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:271: TEST_backfill_priority: sleep 2 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:42.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:42.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:43:42.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:273: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:42.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:274: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:42.901 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:42.902 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:42.903 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:42.904 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:42.905 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:42.905 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:42.905 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:42.905 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:42.905 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:275: TEST_backfill_priority: PRIO=151 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:276: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:42.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2024-09-10T19:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: eval 'ITEM="7.0"' 2024-09-10T19:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:282: TEST_backfill_priority: ITEM=7.0 2024-09-10T19:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:283: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-10T19:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: cat td/osd-backfill-prio/out 2024-09-10T19:43:42.909 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-10T19:43:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:288: TEST_backfill_priority: PRIO=254 2024-09-10T19:43:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:289: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2024-09-10T19:43:42.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:296: TEST_backfill_priority: ceph osd unset noout 2024-09-10T19:43:43.516 INFO:tasks.workunit.client.0.smithi130.stderr:noout is unset 2024-09-10T19:43:43.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:297: TEST_backfill_priority: ceph osd unset nobackfill 2024-09-10T19:43:44.252 INFO:tasks.workunit.client.0.smithi130.stderr:nobackfill is unset 2024-09-10T19:43:44.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:43:44.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:43:44.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:43:44.266 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:43:44.266 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:299: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations' 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations' 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:43:44.267 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:43:44.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:43:44.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:43:44.777 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:43:44.777 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:43:44.777 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:43:44.778 INFO:tasks.workunit.client.0.smithi130.stderr:4' 2024-09-10T19:43:44.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:43:44.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:44.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:43:45.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836542 2024-09-10T19:43:45.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836542 2024-09-10T19:43:45.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542' 2024-09-10T19:43:45.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:45.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:43:45.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673020 2024-09-10T19:43:45.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673020 2024-09-10T19:43:45.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673020' 2024-09-10T19:43:45.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:45.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:43:45.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509497 2024-09-10T19:43:45.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509497 2024-09-10T19:43:45.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673020 2-64424509497' 2024-09-10T19:43:45.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:45.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:43:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345975 2024-09-10T19:43:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345975 2024-09-10T19:43:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673020 2-64424509497 3-85899345975' 2024-09-10T19:43:45.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:43:45.421 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182452 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182452 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836542 1-42949673020 2-64424509497 3-85899345975 4-107374182452' 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836542 2024-09-10T19:43:45.540 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:45.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:43:45.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836542 2024-09-10T19:43:45.542 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:45.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836542 2024-09-10T19:43:45.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836542' 2024-09-10T19:43:45.543 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836542 2024-09-10T19:43:45.543 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:45.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836542 2024-09-10T19:43:45.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:43:46.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:43:46.997 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:43:47.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836542 -lt 21474836542 2024-09-10T19:43:47.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:47.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673020 2024-09-10T19:43:47.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:47.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:43:47.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673020 2024-09-10T19:43:47.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:47.406 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673020 2024-09-10T19:43:47.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673020 2024-09-10T19:43:47.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673020' 2024-09-10T19:43:47.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:43:47.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673020 -lt 42949673020 2024-09-10T19:43:47.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:47.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509497 2024-09-10T19:43:47.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:47.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:43:47.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509497 2024-09-10T19:43:47.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:47.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509497 2024-09-10T19:43:47.888 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509497 2024-09-10T19:43:47.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509497' 2024-09-10T19:43:47.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:43:48.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509498 -lt 64424509497 2024-09-10T19:43:48.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:48.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345975 2024-09-10T19:43:48.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:48.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:43:48.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345975 2024-09-10T19:43:48.285 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:48.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345975 2024-09-10T19:43:48.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345975' 2024-09-10T19:43:48.286 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345975 2024-09-10T19:43:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:43:48.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345975 -lt 85899345975 2024-09-10T19:43:48.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:43:48.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182452 2024-09-10T19:43:48.752 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:43:48.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:43:48.753 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182452 2024-09-10T19:43:48.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:43:48.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182452 2024-09-10T19:43:48.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182452' 2024-09-10T19:43:48.755 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182452 2024-09-10T19:43:48.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:43:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182453 -lt 107374182452 2024-09-10T19:43:49.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:43:49.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:49.292 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 3 == 0 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:49.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:50.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:50.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:50.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:50.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:50.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:50.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T19:43:50.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:43:50.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T19:43:50.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:50.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:50.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:51.004 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:51.004 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:51.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:51.006 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:51.007 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:51.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:51.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:51.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:51.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:51.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:51.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:51.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:43:52.107 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:43:52.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=9970226 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 9970226 '!=' null 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:52.677 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:52.755 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:52.755 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:52.756 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:52.757 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:52.758 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:52.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:52.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:52.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:52.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:53.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:53.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:53.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:53.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:43:53.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:43:54.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=9970226 2024-09-10T19:43:54.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 9970226 '!=' null 2024-09-10T19:43:54.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:43:54.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:54.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:54.558 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:54.635 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:54.636 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:54.637 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:54.638 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:54.639 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:54.639 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:54.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:54.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:54.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:54.745 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:55.127 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:55.127 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:55.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:55.128 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:43:55.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:43:55.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:43:56.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=18541478 2024-09-10T19:43:56.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 18541478 '!=' null 2024-09-10T19:43:56.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:43:56.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:56.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:56.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:56.368 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:56.368 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:56.368 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:56.369 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:56.370 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:56.371 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:56.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:56.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:56.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:56.477 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:56.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:56.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:56.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:56.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:56.478 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:56.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:56.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:56.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:56.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:43:57.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:43:57.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=15725836 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 15725836 '!=' null 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:58.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:58.115 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:58.116 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:58.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:58.118 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:58.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:58.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:58.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:58.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:58.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:58.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:58.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:58.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:43:58.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:43:58.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:43:58.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:43:58.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:43:59.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:43:59.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:43:59.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:43:59.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=15725656 2024-09-10T19:43:59.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 15725656 '!=' null 2024-09-10T19:43:59.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:43:59.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:43:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:43:59.856 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:59.857 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:43:59.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:43:59.859 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:43:59.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:43:59.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:43:59.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:43:59.966 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:00.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:00.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:00.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:00.354 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:00.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=24462205 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 24462205 '!=' null 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:01.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:01.838 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:01.838 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:01.838 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:01.838 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:01.839 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:01.840 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:01.841 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:01.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:01.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:01.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:01.949 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:02.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:02.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:02.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:02.339 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:02.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:02.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:02.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:02.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17473018 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17473018 '!=' null 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:03.589 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:03.666 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:03.667 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:03.668 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:03.669 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:03.669 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:03.669 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:03.669 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:03.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:03.775 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:04.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:04.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:04.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:04.177 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:04.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:04.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:04.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:04.804 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:05.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=26209561 2024-09-10T19:44:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 26209561 '!=' null 2024-09-10T19:44:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:05.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:05.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:05.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:05.501 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:05.501 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:05.501 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:05.501 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:05.501 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:05.502 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:05.503 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:05.504 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:05.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:05.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:05.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:05.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:05.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:05.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:05.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:05.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:05.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:05.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:05.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:05.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:05.996 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:06.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:06.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17473136 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17473136 '!=' null 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:07.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:07.465 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:07.465 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:07.466 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:07.467 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:07.468 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:07.469 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:07.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:07.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:07.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:07.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:07.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:07.955 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:07.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:07.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:08.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:08.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17473357 2024-09-10T19:44:09.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17473357 '!=' null 2024-09-10T19:44:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:09.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:09.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:09.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:09.237 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 151, 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:09.238 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "7.0", 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 254, 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:09.239 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:09.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:09.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:09.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:09.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:09.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:09.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:09.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:10.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=25336696 2024-09-10T19:44:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 25336696 '!=' null 2024-09-10T19:44:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:10.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:10.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:10.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:11.050 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:11.050 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:11.051 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:11.052 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:11.053 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:11.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:11.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:11.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:11.160 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:11.161 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:11.569 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:11.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:11.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:11.570 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:12.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:12.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:12.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=11794601 2024-09-10T19:44:12.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 11794601 '!=' null 2024-09-10T19:44:12.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:12.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:12.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:12.937 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:13.015 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:13.016 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:13.017 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:13.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:13.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:13.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:44:13.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:13.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:13.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:14.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 3 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:14.165 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=22278685 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 22278685 '!=' null 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:14.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:14.832 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:14.833 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:14.834 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:14.835 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:14.835 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:14.835 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:14.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:14.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:14.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:14.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:14.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:14.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:14.942 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:14.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:14.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:15.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:15.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:15.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:15.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:15.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:15.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 0 2024-09-10T19:44:15.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:44:15.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=1 2024-09-10T19:44:15.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:15.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:15.887 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:15.964 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:15.965 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:15.966 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:15.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:16.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:16.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:16.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:16.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:16.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:16.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:16.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:16.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:16.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:16.991 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=22278736 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 22278736 '!=' null 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:17.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:17.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:17.634 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:17.635 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:17.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:17.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:17.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:17.742 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:18.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:18.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:18.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:18.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:18.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:18.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:18.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:18.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:19.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=10484205 2024-09-10T19:44:19.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 10484205 '!=' null 2024-09-10T19:44:19.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:19.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:19.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:19.350 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:19.426 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:19.426 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:19.426 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:19.427 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:19.428 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:19.429 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:19.429 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:19.429 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:19.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:19.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:19.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:19.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:20.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:20.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:20.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:20.001 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:20.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:20.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:20.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:20.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:21.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=19570948 2024-09-10T19:44:21.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 19570948 '!=' null 2024-09-10T19:44:21.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:21.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:21.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:21.275 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:21.352 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:21.353 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:21.354 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:21.354 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:21.354 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:21.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:21.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:21.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:21.461 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:21.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:21.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:21.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:21.850 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:22.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:22.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:22.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:22.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17172545 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17172545 '!=' null 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:23.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:23.214 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:23.215 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:23.216 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:23.217 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:23.217 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:23.217 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:23.217 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:23.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:23.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:23.324 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:23.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:23.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:23.733 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:24.334 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23589357 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23589357 '!=' null 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:24.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:25.004 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:25.005 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:25.006 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:25.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:25.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:25.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:25.113 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:25.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:25.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:25.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:25.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:25.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:25.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:25.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:25.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:25.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:26.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:26.117 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16599819 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16599819 '!=' null 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:26.734 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:26.815 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:26.816 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:26.817 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:26.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:26.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:27.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:27.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:27.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:27.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:28.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:28.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:28.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:28.100 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16599819 2024-09-10T19:44:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16599819 '!=' null 2024-09-10T19:44:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:28.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:28.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:28.795 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:28.795 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:28.795 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:28.795 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:28.796 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:28.797 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:28.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:28.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:28.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:28.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:28.905 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:28.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:28.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:28.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:28.906 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:29.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:29.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:29.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:29.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:29.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:29.825 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:30.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16599772 2024-09-10T19:44:30.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16599772 '!=' null 2024-09-10T19:44:30.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:30.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:30.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:30.377 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:30.454 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:30.454 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:30.454 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:30.455 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:30.456 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:30.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:30.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:30.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:30.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:31.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:31.009 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:31.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:31.010 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:31.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:31.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:31.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:31.790 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=25336452 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 25336452 '!=' null 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:32.341 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:32.420 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:32.421 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:32.422 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:32.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:32.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:32.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:32.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:33.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:33.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:33.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:33.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:33.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:33.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:33.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16599524 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16599524 '!=' null 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:34.235 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:34.314 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:34.315 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:34.316 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:34.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:34.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:34.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:34.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:34.809 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:34.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:34.810 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:35.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:35.411 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:35.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=25336263 2024-09-10T19:44:35.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 25336263 '!=' null 2024-09-10T19:44:35.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:35.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:35.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:35.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:36.070 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:36.070 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:36.070 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [ 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "priority": 110, 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "items": [ 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:44:36.071 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "3.0", 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 151, 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:36.072 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:36.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:36.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:36.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:36.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:36.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:36.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:37.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:37.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:37.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17473223 2024-09-10T19:44:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17473223 '!=' null 2024-09-10T19:44:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:37.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:37.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:37.915 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:37.991 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:37.992 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:37.993 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:37.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:38.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:38.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:38.101 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:38.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:38.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:38.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:38.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:39.176 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:39.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=20967903 2024-09-10T19:44:39.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 20967903 '!=' null 2024-09-10T19:44:39.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:39.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:39.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:39.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:39.894 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:39.894 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:39.894 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:39.894 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:39.895 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:39.896 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:39.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:40.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:40.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:40.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:40.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:40.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:44:40.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:40.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:40.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:40.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 3 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 1 '!=' 1 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:40.921 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:41.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23589278 2024-09-10T19:44:41.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23589278 '!=' null 2024-09-10T19:44:41.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:41.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:41.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:41.689 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:41.689 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:41.689 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:41.690 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:41.691 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:41.692 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:41.692 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:41.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:41.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:41.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:42.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:42.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:42.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:42.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:42.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:42.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 1 2024-09-10T19:44:42.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:44:42.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=2 2024-09-10T19:44:42.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:42.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:42.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:42.834 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:42.835 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:42.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:42.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:42.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:42.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:42.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:42.943 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:42.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:42.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:42.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:43.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:43.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:43.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:43.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:43.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:43.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:43.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:43.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:43.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:43.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:43.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:43.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:43.893 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=10484132 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 10484132 '!=' null 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:44.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:44.577 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:44.577 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:44.577 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:44.578 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:44.579 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:44.579 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:44.579 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:44.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:44.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:44.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:44.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:45.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:45.098 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:45.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:45.099 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:45.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:45.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:46.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=10484263 2024-09-10T19:44:46.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 10484263 '!=' null 2024-09-10T19:44:46.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:46.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:46.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:46.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:46.487 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:46.488 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:46.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:46.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:46.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:46.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:46.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:46.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:46.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:46.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:47.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:47.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:47.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=21313431 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 21313431 '!=' null 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:48.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:48.177 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:48.178 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:48.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:48.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:48.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:48.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:48.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:48.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:48.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:48.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:48.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:49.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:49.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:49.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:49.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:49.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:49.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:49.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:49.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:49.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:50.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=17823154 2024-09-10T19:44:50.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 17823154 '!=' null 2024-09-10T19:44:50.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:50.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:50.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:50.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:50.088 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:50.088 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:50.088 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:50.088 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:50.089 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:50.090 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:50.090 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:50.090 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:50.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:50.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:50.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:50.198 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:50.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:50.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:50.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:50.582 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:51.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:51.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:51.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:51.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=23564742 2024-09-10T19:44:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 23564742 '!=' null 2024-09-10T19:44:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:51.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:51.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:51.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:51.865 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:51.866 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:51.867 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:51.867 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:51.867 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:51.867 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:51.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:51.975 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:51.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:52.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:52.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:52.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:52.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:52.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:53.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=15726237 2024-09-10T19:44:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 15726237 '!=' null 2024-09-10T19:44:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:53.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:53.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:53.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:53.552 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:53.553 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:53.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:53.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:53.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:53.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:53.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:53.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:53.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:53.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:53.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:54.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:54.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:54.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:54.137 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:54.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:55.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=15726277 2024-09-10T19:44:55.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 15726277 '!=' null 2024-09-10T19:44:55.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:55.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:55.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:55.237 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:55.314 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:55.315 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:55.316 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:55.316 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:55.316 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:55.316 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:55.316 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:55.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:55.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:55.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:55.424 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:55.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:55.871 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:55.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:55.872 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:56.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:56.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:57.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=24463044 2024-09-10T19:44:57.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 24463044 '!=' null 2024-09-10T19:44:57.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:57.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:57.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:57.073 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:57.150 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:57.151 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:57.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:57.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:57.259 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:57.260 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:57.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:57.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:57.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:57.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:44:58.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:44:58.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:44:58.346 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16599885 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16599885 '!=' null 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:44:59.006 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:44:59.085 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:44:59.086 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:44:59.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:44:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:44:59.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:44:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:44:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:44:59.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:44:59.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:44:59.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:44:59.195 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:44:59.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:44:59.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:44:59.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:44:59.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:00.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:00.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:00.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:00.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=24463067 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 24463067 '!=' null 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:00.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:00.896 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:00.897 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:00.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:01.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:01.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:01.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:01.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:01.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:01.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:01.389 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:01.988 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:02.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=24463067 2024-09-10T19:45:02.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 24463067 '!=' null 2024-09-10T19:45:02.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:02.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:02.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:02.740 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:02.817 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:02.817 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:02.817 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:02.818 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:02.819 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:02.819 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:02.819 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:02.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:02.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:02.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:02.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:02.926 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:02.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:02.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:02.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:02.927 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:03.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:03.314 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:03.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:03.315 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:03.919 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16600000 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16600000 '!=' null 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:04.528 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [ 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "item": "1.0", 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "prio": 110, 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: "can_preempt": true 2024-09-10T19:45:04.608 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:04.609 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:04.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:04.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:04.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:04.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:04.718 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:05.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:05.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:05.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:05.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:05.827 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16600041 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16600041 '!=' null 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:06.462 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:06.539 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:06.539 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:06.540 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:06.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:06.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:06.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:06.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:06.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:06.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:06.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:06.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:06.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:07.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:07.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:07.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:07.051 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:07.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:07.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:07.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=24900278 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 24900278 '!=' null 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:08.141 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:08.218 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:08.218 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:08.218 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:08.218 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:08.219 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:08.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:08.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:08.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:08.328 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:08.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:08.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:08.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:08.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:09.266 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=13105268 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 13105268 '!=' null 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:09.816 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:09.893 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:09.894 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:09.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:10.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:10.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=2 2024-09-10T19:45:10.385 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:10.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:10.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:10.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 = 3 2024-09-10T19:45:10.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 2 '!=' 2 2024-09-10T19:45:10.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:45:10.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:45:10.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:45:10.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:45:10.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:45:10.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:45:10.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=14415643 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 14415643 '!=' null 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: CEPH_ARGS= 2024-09-10T19:45:11.518 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_recovery_reservations 2024-09-10T19:45:11.594 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:11.594 INFO:tasks.workunit.client.0.smithi130.stdout: "local_reservations": { 2024-09-10T19:45:11.594 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "remote_reservations": { 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "max_allowed": 1, 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "min_priority": 0, 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "queues": [], 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: "in_progress": [] 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:11.595 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:11.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:45:11.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:45:11.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:45:11.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:45:11.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:45:11.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:45:11.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:45:11.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:45:11.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:45:12.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=3 2024-09-10T19:45:12.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:45:12.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:45:12.087 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:45:12.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 = 3 2024-09-10T19:45:12.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:45:12.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:45:12.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:301: TEST_backfill_priority: ceph pg dump pgs 2024-09-10T19:45:13.006 INFO:tasks.workunit.client.0.smithi130.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-10T19:45:13.006 INFO:tasks.workunit.client.0.smithi130.stdout:7.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:44:12.172831+0000 92'150 116:394 [1,4] 1 [1,4] 1 0'0 2024-09-10T19:41:02.660463+0000 0'0 2024-09-10T19:41:02.660463+0000 0 0 periodic scrub scheduled @ 2024-09-12T00:12:50.261637+0000 0 0 2024-09-10T19:45:13.006 INFO:tasks.workunit.client.0.smithi130.stdout:3.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:44:39.391411+0000 92'150 116:429 [1,2] 1 [1,2] 1 0'0 2024-09-10T19:40:52.405381+0000 0'0 2024-09-10T19:40:52.405381+0000 0 0 periodic scrub scheduled @ 2024-09-12T04:19:48.956726+0000 0 0 2024-09-10T19:45:13.006 INFO:tasks.workunit.client.0.smithi130.stdout:1.0 50 0 0 0 0 524288000 0 0 50 100 50 active+clean 2024-09-10T19:45:06.031080+0000 92'150 116:433 [1,2] 1 [1,2] 1 0'0 2024-09-10T19:40:47.465055+0000 0'0 2024-09-10T19:40:47.465055+0000 0 0 periodic scrub scheduled @ 2024-09-11T20:04:56.289042+0000 0 0 2024-09-10T19:45:13.007 INFO:tasks.workunit.client.0.smithi130.stdout: 2024-09-10T19:45:13.007 INFO:tasks.workunit.client.0.smithi130.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-10T19:45:13.007 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:45:13.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: get_asok_path osd.1 2024-09-10T19:45:13.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2024-09-10T19:45:13.018 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2024-09-10T19:45:13.018 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:45:13.018 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:13.019 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:45:13.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.27247/ceph-osd.1.asok 2024-09-10T19:45:13.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: CEPH_ARGS= 2024-09-10T19:45:13.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:303: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.27247/ceph-osd.1.asok dump_pgstate_history 2024-09-10T19:45:13.097 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:45:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "pgs": [ 2024-09-10T19:45:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "7.0", 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "60", 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:02.853273+0000", 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:02.853281+0000" 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:02.853286+0000", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:03.004305+0000" 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:03.004306+0000", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:03.004387+0000" 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:03.004388+0000", 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:03.004405+0000" 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:03.004405+0000", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:03.264238+0000" 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:02.853284+0000", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:03.264243+0000" 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:03.264414+0000", 2024-09-10T19:45:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.463802+0000" 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.463803+0000", 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.463877+0000" 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.463877+0000", 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881214+0000" 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:03.264244+0000", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881233+0000" 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:02.853281+0000", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881289+0000" 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:02.853270+0000", 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881294+0000" 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "60", 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881295+0000", 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881453+0000" 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "90", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881458+0000", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.881467+0000" 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881473+0000", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.998849+0000" 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.998850+0000", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.998905+0000" 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.998905+0000", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:04.998917+0000" 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.998917+0000", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:05.459487+0000" 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881470+0000", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:05.459500+0000" 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:05.459614+0000", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:05.929308+0000" 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:05.929309+0000", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:05.929349+0000" 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:05.929350+0000", 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803812+0000" 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:05.459501+0000", 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803833+0000" 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881468+0000", 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803863+0000" 2024-09-10T19:45:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:04.881454+0000", 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803866+0000" 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "90", 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.803866+0000", 2024-09-10T19:45:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804011+0000" 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "97", 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804015+0000", 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804023+0000" 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804028+0000", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804111+0000" 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804111+0000", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804158+0000" 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804158+0000", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804173+0000" 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804174+0000", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.505143+0000" 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804026+0000", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.505155+0000" 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.505254+0000", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.563975+0000" 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.563977+0000", 2024-09-10T19:45:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.564012+0000" 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.564012+0000", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709322+0000" 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.505155+0000", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709337+0000" 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804023+0000", 2024-09-10T19:45:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709403+0000" 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804012+0000", 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709406+0000" 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "97", 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709407+0000", 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709664+0000" 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "98", 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709668+0000", 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.709680+0000" 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709687+0000", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.818536+0000" 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.818537+0000", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.818588+0000" 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709683+0000", 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:53.818595+0000" 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.818595+0000", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.052911+0000" 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709680+0000", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.052948+0000" 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:53.709664+0000", 2024-09-10T19:45:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.052955+0000" 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "98", 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.052956+0000", 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.053268+0000" 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "111", 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.053275+0000", 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.053295+0000" 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.053309+0000", 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.187379+0000" 2024-09-10T19:45:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.187379+0000", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.187444+0000" 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.187444+0000", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.187458+0000" 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.187459+0000", 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.522510+0000" 2024-09-10T19:45:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.053303+0000", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.522525+0000" 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.522790+0000", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.749561+0000" 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.749562+0000", 2024-09-10T19:45:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.749694+0000" 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.749694+0000", 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:42:54.750110+0000" 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.750111+0000", 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:29.584011+0000" 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/NotBackfilling", 2024-09-10T19:45:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:29.584011+0000", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:29.584153+0000" 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:29.584154+0000", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:40.684921+0000" 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:40.684922+0000", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:40.685387+0000" 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:40.685387+0000", 2024-09-10T19:45:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:10.609860+0000" 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:10.609862+0000", 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:10.609977+0000" 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:10.609978+0000", 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:11.050624+0000" 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.522526+0000", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:11.050651+0000" 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.053296+0000", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:11.050697+0000" 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:42:54.053268+0000", 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:11.050700+0000" 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "111", 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:11.050701+0000", 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:11.050992+0000" 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "3.0", 2024-09-10T19:45:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "38", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.770748+0000", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:52.770755+0000" 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.770760+0000", 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:52.771569+0000" 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.771570+0000", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:52.771617+0000" 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.771617+0000", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:52.771625+0000" 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.771625+0000", 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:53.298716+0000" 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.770757+0000", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:53.298722+0000" 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:53.298858+0000", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.111137+0000" 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.111138+0000", 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.111381+0000" 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.111381+0000", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.408774+0000" 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:53.298724+0000", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.408797+0000" 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.770755+0000", 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.408850+0000" 2024-09-10T19:45:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:52.770744+0000", 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.408853+0000" 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "38", 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.408855+0000", 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.409016+0000" 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "88", 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.409019+0000", 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.409029+0000" 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.409034+0000", 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.551002+0000" 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.551002+0000", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.551081+0000" 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.551081+0000", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.551092+0000" 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.551093+0000", 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.974899+0000" 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.409031+0000", 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:54.974913+0000" 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.975052+0000", 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:55.075969+0000" 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:55.075970+0000", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:55.076027+0000" 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:55.076027+0000", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803791+0000" 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.974914+0000", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803824+0000" 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.409029+0000", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803860+0000" 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:54.409016+0000", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.803863+0000" 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "90", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.803864+0000", 2024-09-10T19:45:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804052+0000" 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "103", 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804055+0000", 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.804063+0000" 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804068+0000", 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.935266+0000" 2024-09-10T19:45:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.935266+0000", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.935312+0000" 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.935313+0000", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:33.935322+0000" 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.935322+0000", 2024-09-10T19:45:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.505236+0000" 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804066+0000", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.505243+0000" 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.505291+0000", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.642533+0000" 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.642534+0000", 2024-09-10T19:45:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:41:34.642569+0000" 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.642569+0000", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878296+0000" 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:34.505243+0000", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878335+0000" 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804063+0000", 2024-09-10T19:45:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878447+0000" 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:41:33.804052+0000", 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878453+0000" 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "104", 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878453+0000", 2024-09-10T19:45:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878853+0000" 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "105", 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878857+0000", 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.878868+0000" 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878881+0000", 2024-09-10T19:45:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.879388+0000" 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.879389+0000", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.879477+0000" 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878875+0000", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:14.879489+0000" 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.879490+0000", 2024-09-10T19:45:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.106078+0000" 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878869+0000", 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.106104+0000" 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:14.878853+0000", 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.106108+0000" 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "105", 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106108+0000", 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.106282+0000" 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "113", 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106286+0000", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.106297+0000" 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106304+0000", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.230531+0000" 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.230531+0000", 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.230598+0000" 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.230599+0000", 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.230613+0000" 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.230614+0000", 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.515396+0000" 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106300+0000", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:15.515410+0000" 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.515646+0000", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:16.510756+0000" 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:16.510757+0000", 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:29.583990+0000" 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:29.583991+0000", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:29.584381+0000" 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:29.584382+0000", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:40.684949+0000" 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/NotBackfilling", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:40.684950+0000", 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:40.685097+0000" 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:40.685097+0000", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:10.609870+0000" 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:10.609871+0000", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:10.610361+0000" 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:10.610361+0000", 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:37.916920+0000" 2024-09-10T19:45:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:37.916921+0000", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:37.917046+0000" 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:37.917047+0000", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:38.359930+0000" 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.515411+0000", 2024-09-10T19:45:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:38.359956+0000" 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106297+0000", 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:38.360012+0000" 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:15.106282+0000", 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:38.360016+0000" 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "113", 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:38.360017+0000", 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:38.360271+0000" 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "pg": "1.0", 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "history": [ 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "27", 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Initial", 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136066+0000", 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.136165+0000" 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "27", 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136168+0000", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.136192+0000" 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "29", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136195+0000", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.136201+0000" 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136206+0000", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.170240+0000" 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.170241+0000", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.170294+0000" 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.170295+0000", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.170303+0000" 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.170304+0000", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.471660+0000" 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136204+0000", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:48.471668+0000" 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.471856+0000", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279264+0000" 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.471669+0000", 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279279+0000" 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136202+0000", 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279295+0000" 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:48.136192+0000", 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279299+0000" 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "29", 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279300+0000", 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279442+0000" 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "100", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279445+0000", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.279452+0000" 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279457+0000", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.368085+0000" 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.368086+0000", 2024-09-10T19:45:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.368137+0000" 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.368138+0000", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.368147+0000" 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.368148+0000", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.594003+0000" 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279454+0000", 2024-09-10T19:45:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.594022+0000" 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.594201+0000", 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.714487+0000" 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.714488+0000", 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:40:49.714544+0000" 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.714545+0000", 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.614996+0000" 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.594023+0000", 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.615023+0000" 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279452+0000", 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.615113+0000" 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:40:49.279443+0000", 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.615120+0000" 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "100", 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615120+0000", 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.615499+0000" 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "101", 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615506+0000", 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.615524+0000" 2024-09-10T19:45:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615536+0000", 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.616072+0000" 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.616073+0000", 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.616136+0000" 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615531+0000", 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.616144+0000" 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.616144+0000", 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.822819+0000" 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615524+0000", 2024-09-10T19:45:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.822841+0000" 2024-09-10T19:45:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.615500+0000", 2024-09-10T19:45:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.822845+0000" 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "101", 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.822845+0000", 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.823034+0000" 2024-09-10T19:45:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "115", 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Start", 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.823038+0000", 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.823050+0000" 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.823058+0000", 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.967649+0000" 2024-09-10T19:45:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.967650+0000", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.967766+0000" 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.967766+0000", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:05.967802+0000" 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.967802+0000", 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:06.226673+0000" 2024-09-10T19:45:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Peering", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.823054+0000", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:06.226681+0000" 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Activating", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:06.226967+0000", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:43:06.570873+0000" 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:06.570874+0000", 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:37.916921+0000" 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:37.916922+0000", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:44:37.917326+0000" 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Backfilling", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:44:37.917329+0000", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.160576+0000" 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:45:05.160578+0000", 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.160700+0000" 2024-09-10T19:45:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active/Clean", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:45:05.160700+0000", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.425949+0000" 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary/Active", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:06.226682+0000", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.425981+0000" 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started/Primary", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.823050+0000", 2024-09-10T19:45:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.426052+0000" 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Started", 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:43:05.823034+0000", 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.426062+0000" 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "epoch": "115", 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "states": [ 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "state": "Reset", 2024-09-10T19:45:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "enter": "2024-09-10T19:45:05.426063+0000", 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "exit": "2024-09-10T19:45:05.426498+0000" 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:45:13.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:305: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2024-09-10T19:45:13.176 INFO:tasks.workunit.client.0.smithi130.stdout:TEST PASSED 2024-09-10T19:45:13.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:309: TEST_backfill_priority: echo TEST PASSED 2024-09-10T19:45:13.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:312: TEST_backfill_priority: delete_pool test1 2024-09-10T19:45:13.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test1 2024-09-10T19:45:13.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-10T19:45:13.579 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test1' does not exist 2024-09-10T19:45:13.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:313: TEST_backfill_priority: delete_pool test3 2024-09-10T19:45:13.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test3 2024-09-10T19:45:13.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-10T19:45:14.277 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test3' does not exist 2024-09-10T19:45:14.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:314: TEST_backfill_priority: delete_pool test7 2024-09-10T19:45:14.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test7 2024-09-10T19:45:14.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-09-10T19:45:14.964 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test7' does not exist 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:315: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:45:14.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:316: TEST_backfill_priority: return 0 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-prio.sh:43: run: teardown td/osd-backfill-prio 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-prio 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-prio KILL 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:45:20.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:45:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:45:20.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:45:20.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:45:20.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:45:20.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:45:20.300 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:45:20.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:45:20.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:45:20.302 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:45:20.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:45:20.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:45:20.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:45:20.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:45:20.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-prio 2024-09-10T19:45:20.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:45:20.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:20.340 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:45:20.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-backfill-prio 0 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-prio 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-prio KILL 2024-09-10T19:45:20.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:45:20.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:45:20.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:45:20.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:45:20.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:45:20.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:45:20.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:45:20.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:45:20.345 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:45:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:45:20.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:45:20.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:45:20.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:45:20.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:45:20.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:45:20.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T19:45:20.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-prio 2024-09-10T19:45:20.351 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:45:20.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:20.352 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.27247 2024-09-10T19:45:20.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.27247 2024-09-10T19:45:20.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:45:20.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:45:20.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T19:45:20.354 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T19:45:20.354 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T19:45:20.423 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-recovery-log.sh... 2024-09-10T19:45:20.423 DEBUG:teuthology.orchestra.run.smithi130:workunit test osd-backfill/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh 2024-09-10T19:45:20.486 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T19:45:20.491 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T19:45:20.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T19:45:20.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T19:45:20.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T19:45:20.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-backfill-recovery-log 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:22: run: shift 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:27: run: uuidgen 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none ' 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2024-09-10T19:45:20.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-10T19:45:20.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: set 2024-09-10T19:45:20.494 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T19:45:20.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:31: run: local 'funcs=TEST_backfill_log_1 2024-09-10T19:45:20.494 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_log_2 2024-09-10T19:45:20.494 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_recovery_1 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_recovery_2' 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: for func in $funcs 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: setup td/osd-backfill-recovery-log 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-recovery-log 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:45:20.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:45:20.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:45:20.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:45:20.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:45:20.496 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:45:20.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:45:20.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:45:20.498 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:45:20.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:45:20.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:45:20.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:45:20.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:45:20.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:45:20.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:45:20.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:45:20.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-recovery-log 2024-09-10T19:45:20.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:45:20.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:20.504 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:20.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53443 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:106: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:108: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:45:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local loglen=2 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local dupslen=8 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local objects=150 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:46: _common_test: local moreobjects=0 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:48: _common_test: local OSDS=6 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mon td/osd-backfill-recovery-log a 2024-09-10T19:45:20.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:45:20.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:45:20.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:45:20.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-recovery-log/a 2024-09-10T19:45:20.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:20.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:45:21.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:21.138 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:45:21.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:45:21.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get fsid 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:45:21.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get mon_host 2024-09-10T19:45:21.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: run_mgr td/osd-backfill-recovery-log x 2024-09-10T19:45:21.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:21.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:45:21.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:45:21.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:45:21.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-recovery-log/x 2024-09-10T19:45:21.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:22.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:22.151 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:45:22.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:45:22.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export CEPH_ARGS 2024-09-10T19:45:22.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:45:22.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:45:22.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: expr 6 - 1 2024-09-10T19:45:22.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: seq 0 5 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 0 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2024-09-10T19:45:22.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:45:22.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:45:22.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2 2024-09-10T19:45:22.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2' 2024-09-10T19:45:22.190 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2 2024-09-10T19:45:22.190 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:45:22.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSoeBm2RNEDBAALRoaNdjebkNXF1UXw8omeA== 2024-09-10T19:45:22.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSoeBm2RNEDBAALRoaNdjebkNXF1UXw8omeA=="}' 2024-09-10T19:45:22.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2 -i td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:45:22.480 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:45:22.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:45:22.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSoeBm2RNEDBAALRoaNdjebkNXF1UXw8omeA== --osd-uuid 5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2 2024-09-10T19:45:22.529 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:22.527+0000 7f657ddbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:22.557 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:22.555+0000 7f657ddbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:22.558 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:22.556+0000 7f657ddbf640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:22.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:22.557+0000 7f657ddbf640 -1 bdev(0x556246c47c00 td/osd-backfill-recovery-log/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:45:22.559 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:22.557+0000 7f657ddbf640 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2024-09-10T19:45:28.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2024-09-10T19:45:28.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:45:28.108 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:45:28.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:45:28.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:45:28.656 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:45:28.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:45:28.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:45:28.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:45:28.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:45:28.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:45:28.687 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:28.685+0000 7fe036813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:28.694 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:28.692+0000 7fe036813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:28.695 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:28.694+0000 7fe036813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:29.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:29.093 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:45:29.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:45:29.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:29.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:29.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:29.540 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:29.538+0000 7fe036813640 -1 Falling back to public interface 2024-09-10T19:45:30.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:30.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:30.536 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:45:30.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:45:30.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:30.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:30.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:31.497 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:31.495+0000 7fe036813640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:45:31.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:31.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:31.996 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:45:31.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:45:31.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:31.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:32.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:33.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:33.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:33.527 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:45:33.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:45:33.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:33.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:33.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:34.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:34.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:34.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:45:34.965 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:45:34.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:34.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:35.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:36.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:36.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:36.375 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:45:36.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:45:36.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:36.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:36.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:37.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:37.786 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:37.786 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:45:37.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:45:37.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:37.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:38.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:39.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:39.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:39.278 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:45:39.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:45:39.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:39.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:40.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:40.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:40.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:45:40.690 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:45:40.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:40.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:41.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:42.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:42.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:42.088 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:45:42.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:45:42.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:42.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:42.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:43.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:43.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:43.686 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:45:43.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:45:43.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:43.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:44.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:45.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:45.191 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:45:45.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:45.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:45:45.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:45.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:45.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:46.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:46.665 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:45:46.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:46.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:45:46.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:46.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:47.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:48.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:48.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:48.317 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:45:48.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:45:48.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:48.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:49.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:50.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:50.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:50.174 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:45:50.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:45:50.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:50.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:50.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:51.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:51.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:51.638 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:45:51.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:45:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:52.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:45:53.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:45:53.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:45:53.317 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:45:53.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:45:53.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:45:53.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:45:53.772 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2434087158,v1:127.0.0.1:6803/2434087158] [v2:127.0.0.1:6804/2434087158,v1:127.0.0.1:6805/2434087158] exists,up 5b3bbab5-9a0b-42a5-9f0b-4a1e0c41fcd2 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 1 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:45:53.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:45:53.777 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:45:53.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a2d32df8-53de-47ae-ac6a-97d660edae0f 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a2d32df8-53de-47ae-ac6a-97d660edae0f' 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 a2d32df8-53de-47ae-ac6a-97d660edae0f 2024-09-10T19:45:53.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:45:53.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxoeBmsC5RLxAAeNJ9hPrHZAxwvM/4Y+ktOA== 2024-09-10T19:45:53.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxoeBmsC5RLxAAeNJ9hPrHZAxwvM/4Y+ktOA=="}' 2024-09-10T19:45:53.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a2d32df8-53de-47ae-ac6a-97d660edae0f -i td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:45:54.267 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:45:54.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:45:54.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxoeBmsC5RLxAAeNJ9hPrHZAxwvM/4Y+ktOA== --osd-uuid a2d32df8-53de-47ae-ac6a-97d660edae0f 2024-09-10T19:45:54.314 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:54.312+0000 7f5ea6e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:54.328 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:54.325+0000 7f5ea6e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:54.329 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:54.327+0000 7f5ea6e1c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:45:54.329 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:54.327+0000 7f5ea6e1c640 -1 bdev(0x55c17d9d9c00 td/osd-backfill-recovery-log/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:45:54.329 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:45:54.327+0000 7f5ea6e1c640 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2024-09-10T19:46:00.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2024-09-10T19:46:00.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:46:00.045 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:46:00.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:46:00.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:46:00.611 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:46:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:46:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:46:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:46:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:46:00.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:46:00.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:00.641+0000 7f10c12b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:00.650 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:00.648+0000 7f10c12b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:00.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:00.649+0000 7f10c12b6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:01.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:01.065 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:46:01.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:46:01.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:01.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:01.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:01.995 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:01.993+0000 7f10c12b6640 -1 Falling back to public interface 2024-09-10T19:46:02.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:02.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:02.483 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:46:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:46:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:02.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:02.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:03.594 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:03.592+0000 7f10c12b6640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:46:03.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:03.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:03.910 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:46:03.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:46:03.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:03.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:04.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:05.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:05.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:05.488 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:46:05.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:46:05.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:05.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:05.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:06.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:06.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:06.882 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:46:06.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:46:06.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:06.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:07.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:08.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:08.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:08.396 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:46:08.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:46:08.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:08.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:08.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:09.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:09.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:09.927 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:46:09.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:46:09.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:09.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:10.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:11.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:11.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:11.526 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:46:11.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:46:11.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:11.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:11.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:12.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:12.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:12.955 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:46:12.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:46:12.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:12.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:13.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:14.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:14.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:14.454 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:46:14.454 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:46:14.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:14.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:14.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:15.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:15.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:15.950 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:46:15.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:46:15.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:15.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:16.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:17.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:17.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:17.394 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:46:17.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:46:17.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:17.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:18.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:19.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:19.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:19.057 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:46:19.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:46:19.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:19.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:19.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:20.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:20.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:20.793 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:46:20.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:46:20.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:20.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:21.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:22.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:22.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:22.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:46:22.776 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:46:22.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:22.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:23.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:24.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:24.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:24.451 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:46:24.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:46:24.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:24.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:25.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:26.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:26.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:26.481 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:46:26.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:46:26.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:26.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:46:26.924 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2943328536,v1:127.0.0.1:6811/2943328536] [v2:127.0.0.1:6812/2943328536,v1:127.0.0.1:6813/2943328536] exists,up a2d32df8-53de-47ae-ac6a-97d660edae0f 2024-09-10T19:46:26.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:46:26.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:46:26.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:46:26.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 2 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2024-09-10T19:46:26.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:46:26.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca 2024-09-10T19:46:26.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca' 2024-09-10T19:46:26.932 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca 2024-09-10T19:46:26.932 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:46:26.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQASouBmT81mOBAA0vuAJhxXbmYfJA8pN/KjGg== 2024-09-10T19:46:26.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQASouBmT81mOBAA0vuAJhxXbmYfJA8pN/KjGg=="}' 2024-09-10T19:46:26.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca -i td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:46:27.398 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:46:27.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:46:27.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQASouBmT81mOBAA0vuAJhxXbmYfJA8pN/KjGg== --osd-uuid 17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca 2024-09-10T19:46:27.445 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:27.443+0000 7fb7069dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:27.460 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:27.458+0000 7fb7069dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:27.459+0000 7fb7069dd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:27.460+0000 7fb7069dd640 -1 bdev(0x562f424afc00 td/osd-backfill-recovery-log/2/block) open stat got: (1) Operation not permitted 2024-09-10T19:46:27.462 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:27.460+0000 7fb7069dd640 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2024-09-10T19:46:34.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2024-09-10T19:46:34.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:46:34.102 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T19:46:34.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T19:46:34.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:46:34.645 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T19:46:34.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T19:46:34.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:46:34.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:46:34.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:46:34.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:46:34.677 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:34.674+0000 7fd58d60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:34.680 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:34.676+0000 7fd58d60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:34.681 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:34.678+0000 7fd58d60b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:46:35.086 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:46:35.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:35.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:35.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:35.775 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:35.772+0000 7fd58d60b640 -1 Falling back to public interface 2024-09-10T19:46:36.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:36.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:36.504 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:46:36.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:46:36.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:36.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:36.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:37.616 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:46:37.613+0000 7fd58d60b640 -1 osd.2 0 log_to_monitors true 2024-09-10T19:46:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:37.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:37.932 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:46:37.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:46:37.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:37.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:38.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:39.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:39.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:39.551 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:46:39.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:46:39.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:39.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:39.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:40.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:40.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:40.945 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:46:40.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:46:40.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:40.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:41.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:42.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:42.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:42.438 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:46:42.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:46:42.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:42.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:42.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:43.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:43.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:43.873 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:46:43.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:46:43.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:43.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:44.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:45.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:45.379 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:46:45.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:45.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:46:45.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:45.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:45.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:46.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:46.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:46.809 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:46:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:46:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:47.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:48.243 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:46:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:46:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:48.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:48.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:49.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:49.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:49.718 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:46:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:46:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:49.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:50.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:51.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:51.236 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:46:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:46:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:51.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:52.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:53.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:53.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:53.334 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:46:53.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:46:53.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:53.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:54.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:55.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:55.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:55.094 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:46:55.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:46:55.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:55.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:55.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:56.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:56.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:56.933 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:46:56.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:46:56.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:56.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:57.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:46:58.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:46:58.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:46:58.769 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:46:58.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:46:58.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:46:58.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:46:59.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:00.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:00.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:00.511 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:47:00.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:47:00.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:00.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:47:00.964 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3691046673,v1:127.0.0.1:6819/3691046673] [v2:127.0.0.1:6820/3691046673,v1:127.0.0.1:6821/3691046673] exists,up 17cb2de8-28b5-4a1f-9336-c8cdc1e8f6ca 2024-09-10T19:47:00.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 3 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:47:00.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:47:00.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:47:00.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6b668682-8fa7-4c00-94f4-99bbd314ee7f 2024-09-10T19:47:00.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 6b668682-8fa7-4c00-94f4-99bbd314ee7f' 2024-09-10T19:47:00.971 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 6b668682-8fa7-4c00-94f4-99bbd314ee7f 2024-09-10T19:47:00.971 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:47:00.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0ouBmJffEOhAAmDWBAMa4W6hSIBs/PybCHw== 2024-09-10T19:47:00.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0ouBmJffEOhAAmDWBAMa4W6hSIBs/PybCHw=="}' 2024-09-10T19:47:00.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6b668682-8fa7-4c00-94f4-99bbd314ee7f -i td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:47:01.503 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:47:01.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:47:01.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0ouBmJffEOhAAmDWBAMa4W6hSIBs/PybCHw== --osd-uuid 6b668682-8fa7-4c00-94f4-99bbd314ee7f 2024-09-10T19:47:01.550 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:01.548+0000 7feaf0c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:01.555 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:01.553+0000 7feaf0c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:01.556 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:01.554+0000 7feaf0c15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:01.556 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:01.554+0000 7feaf0c15640 -1 bdev(0x55b55fe0bc00 td/osd-backfill-recovery-log/3/block) open stat got: (1) Operation not permitted 2024-09-10T19:47:01.556 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:01.554+0000 7feaf0c15640 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2024-09-10T19:47:07.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2024-09-10T19:47:07.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:47:07.619 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-10T19:47:07.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T19:47:07.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:47:08.159 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-10T19:47:08.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T19:47:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:47:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:47:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:47:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:47:08.191 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:08.189+0000 7f9fbd462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:08.204 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:08.200+0000 7f9fbd462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:08.205 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:08.202+0000 7f9fbd462640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:08.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:08.635 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:47:08.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:47:08.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:08.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:09.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:09.052 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:09.050+0000 7f9fbd462640 -1 Falling back to public interface 2024-09-10T19:47:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:10.020 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:47:10.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:47:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:10.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:10.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:10.823 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:10.821+0000 7f9fbd462640 -1 osd.3 0 log_to_monitors true 2024-09-10T19:47:11.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:11.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:11.431 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:47:11.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:47:11.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:11.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:11.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:12.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:12.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:12.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:47:12.880 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:47:12.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:12.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:13.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:14.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:14.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:14.301 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:47:14.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:47:14.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:14.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:14.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:15.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:15.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:15.716 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:47:15.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:47:15.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:15.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:16.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:17.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:17.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:17.115 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:47:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:47:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:17.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:18.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:18.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:18.614 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:47:18.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:47:18.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:18.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:19.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:20.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:20.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:20.013 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:47:20.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:47:20.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:20.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:20.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:21.528 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:47:21.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:47:21.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:21.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:21.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:22.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:22.934 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:47:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:47:22.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:22.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:23.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:24.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:24.371 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:47:24.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:47:24.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:24.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:25.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:26.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:26.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:26.386 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:47:26.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:47:26.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:26.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:26.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:27.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:27.870 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:47:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:47:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:27.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:28.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:29.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:29.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:29.466 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:47:29.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:47:29.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:29.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:30.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:31.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:31.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:31.184 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:47:31.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:47:31.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:31.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:32.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:33.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:33.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:33.306 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:47:33.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:47:33.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:33.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:33.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:34.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:34.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:34.916 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:47:34.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:47:34.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:34.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:47:35.356 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/681045666,v1:127.0.0.1:6827/681045666] [v2:127.0.0.1:6828/681045666,v1:127.0.0.1:6829/681045666] exists,up 6b668682-8fa7-4c00-94f4-99bbd314ee7f 2024-09-10T19:47:35.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 4 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:47:35.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:47:35.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:47:35.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b63abfd9-3d0e-4ce2-9683-cc58bff1b17a 2024-09-10T19:47:35.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 b63abfd9-3d0e-4ce2-9683-cc58bff1b17a' 2024-09-10T19:47:35.363 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 b63abfd9-3d0e-4ce2-9683-cc58bff1b17a 2024-09-10T19:47:35.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:47:35.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBXouBmeMeEFhAAcJ4MfSiNaxVIsUbIvxrkaw== 2024-09-10T19:47:35.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBXouBmeMeEFhAAcJ4MfSiNaxVIsUbIvxrkaw=="}' 2024-09-10T19:47:35.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b63abfd9-3d0e-4ce2-9683-cc58bff1b17a -i td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:47:35.862 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:47:35.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:47:35.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXouBmeMeEFhAAcJ4MfSiNaxVIsUbIvxrkaw== --osd-uuid b63abfd9-3d0e-4ce2-9683-cc58bff1b17a 2024-09-10T19:47:35.907 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:35.905+0000 7f1fcfc40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:35.922 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:35.920+0000 7f1fcfc40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:35.923 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:35.921+0000 7f1fcfc40640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:35.923 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:35.921+0000 7f1fcfc40640 -1 bdev(0x55597b31dc00 td/osd-backfill-recovery-log/4/block) open stat got: (1) Operation not permitted 2024-09-10T19:47:35.923 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:35.921+0000 7f1fcfc40640 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2024-09-10T19:47:42.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2024-09-10T19:47:42.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:47:42.078 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-10T19:47:42.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T19:47:42.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:47:42.637 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-10T19:47:42.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T19:47:42.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:47:42.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:47:42.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:47:42.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:47:42.669 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:42.667+0000 7fc6eb7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:42.671 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:42.669+0000 7fc6eb7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:42.672 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:42.670+0000 7fc6eb7fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:47:43.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T19:47:43.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:47:43.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T19:47:43.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:47:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:47:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:43.084 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:47:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:47:43.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:43.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:43.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:44.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:44.267+0000 7fc6eb7fd640 -1 Falling back to public interface 2024-09-10T19:47:44.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:44.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:44.511 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:47:44.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:47:44.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:44.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:44.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:45.865 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:47:45.863+0000 7fc6eb7fd640 -1 osd.4 0 log_to_monitors true 2024-09-10T19:47:45.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:45.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:45.946 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:47:45.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:47:45.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:45.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:46.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:47.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:47.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:47.526 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:47:47.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:47:47.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:47.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:48.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:49.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:49.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:49.064 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:47:49.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:47:49.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:49.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:49.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:50.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:50.464 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:47:50.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:50.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:47:50.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:50.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:50.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:51.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:51.978 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:51.979 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:47:51.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:47:51.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:51.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:52.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:53.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:53.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:53.385 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:47:53.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:47:53.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:53.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:53.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:54.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:54.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:54.783 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:47:54.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:47:54.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:54.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:55.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:56.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:56.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:56.198 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:47:56.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:47:56.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:56.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:56.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:57.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:57.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:57.662 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:47:57.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:47:57.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:57.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:47:58.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:47:59.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:47:59.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:47:59.384 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:47:59.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:47:59.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:47:59.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:00.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:01.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:01.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:01.113 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:48:01.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:48:01.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:01.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:01.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:02.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:02.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:02.907 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:48:02.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:48:02.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:02.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:03.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:04.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:04.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:04.433 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:48:04.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:48:04.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:04.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:05.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:06.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:06.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:06.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:48:06.557 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:48:06.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:06.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:07.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:08.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:08.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:08.302 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:48:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:48:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:08.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:48:08.729 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2203655409,v1:127.0.0.1:6835/2203655409] [v2:127.0.0.1:6836/2203655409,v1:127.0.0.1:6837/2203655409] exists,up b63abfd9-3d0e-4ce2-9683-cc58bff1b17a 2024-09-10T19:48:08.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:48:08.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:48:08.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 5 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:48:08.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:48:08.735 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:48:08.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:48:08.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2024-09-10T19:48:08.737 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:48:08.737 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 250c5929-9b02-4997-ac21-b00244a9cf62 2024-09-10T19:48:08.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=250c5929-9b02-4997-ac21-b00244a9cf62 2024-09-10T19:48:08.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 250c5929-9b02-4997-ac21-b00244a9cf62' 2024-09-10T19:48:08.738 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:48:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB4ouBm0fvOLBAAMUdoS+a/LEsT4trTjIF3Ww== 2024-09-10T19:48:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB4ouBm0fvOLBAAMUdoS+a/LEsT4trTjIF3Ww=="}' 2024-09-10T19:48:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 250c5929-9b02-4997-ac21-b00244a9cf62 -i td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:48:09.296 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:48:09.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:48:09.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4ouBm0fvOLBAAMUdoS+a/LEsT4trTjIF3Ww== --osd-uuid 250c5929-9b02-4997-ac21-b00244a9cf62 2024-09-10T19:48:09.343 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:09.341+0000 7fc37a9fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:09.345 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:09.343+0000 7fc37a9fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:09.346 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:09.344+0000 7fc37a9fd640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:09.347 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:09.345+0000 7fc37a9fd640 -1 bdev(0x556c98213c00 td/osd-backfill-recovery-log/5/block) open stat got: (1) Operation not permitted 2024-09-10T19:48:09.347 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:09.345+0000 7fc37a9fd640 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2024-09-10T19:48:15.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2024-09-10T19:48:15.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:48:15.340 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-10T19:48:15.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T19:48:15.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:48:15.880 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-10T19:48:15.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T19:48:15.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:48:15.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:48:15.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:48:15.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:48:15.913 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:15.910+0000 7f7367833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:15.929 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:15.926+0000 7f7367833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:15.930 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:15.927+0000 7f7367833640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:48:16.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:48:16.361 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:48:16.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:16.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:16.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:17.534 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:17.531+0000 7f7367833640 -1 Falling back to public interface 2024-09-10T19:48:17.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:17.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:17.787 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:48:17.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:48:17.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:17.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:18.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:19.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:19.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:19.188 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:48:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:48:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:19.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:19.207 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:48:19.204+0000 7f7367833640 -1 osd.5 0 log_to_monitors true 2024-09-10T19:48:19.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:20.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:20.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:20.727 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:48:20.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:48:20.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:20.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:21.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:22.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:22.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:22.232 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:48:22.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:48:22.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:22.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:22.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:23.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:23.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:23.732 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:48:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:48:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:24.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:25.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:48:25.189 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:48:25.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:25.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:25.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:26.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:26.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:26.630 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:48:26.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:48:26.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:26.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:27.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:28.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:28.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:28.073 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:48:28.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:48:28.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:28.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:28.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:29.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:29.578 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:48:29.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:29.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:48:29.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:29.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:30.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:31.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:31.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:31.051 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:48:31.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:48:31.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:31.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:31.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:32.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:32.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:32.581 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:48:32.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:48:32.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:32.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:33.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:34.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:34.820 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:48:34.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:48:34.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:34.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:35.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:36.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:36.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:36.321 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:48:36.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:48:36.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:36.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:37.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:38.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:38.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:38.124 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:48:38.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:48:38.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:38.125 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:38.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:39.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:39.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:39.713 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:48:39.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:48:39.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:39.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:40.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:41.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:41.354 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:48:41.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:48:41.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:41.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:41.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:48:42.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:48:42.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:48:42.953 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:48:42.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:48:42.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:48:42.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:48:43.365 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/776597905,v1:127.0.0.1:6843/776597905] [v2:127.0.0.1:6844/776597905,v1:127.0.0.1:6845/776597905] exists,up 250c5929-9b02-4997-ac21-b00244a9cf62 2024-09-10T19:48:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:48:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:48:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:48:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:60: _common_test: create_pool test 1 1 2024-09-10T19:48:43.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T19:48:44.205 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2024-09-10T19:48:44.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:48:45.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: seq 1 150 2024-09-10T19:48:45.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:45.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-1 /etc/passwd 2024-09-10T19:48:45.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:45.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-2 /etc/passwd 2024-09-10T19:48:45.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:45.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-3 /etc/passwd 2024-09-10T19:48:45.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:45.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-4 /etc/passwd 2024-09-10T19:48:45.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:45.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-5 /etc/passwd 2024-09-10T19:48:46.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-6 /etc/passwd 2024-09-10T19:48:46.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-7 /etc/passwd 2024-09-10T19:48:46.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-8 /etc/passwd 2024-09-10T19:48:46.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-9 /etc/passwd 2024-09-10T19:48:46.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-10 /etc/passwd 2024-09-10T19:48:46.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-11 /etc/passwd 2024-09-10T19:48:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-12 /etc/passwd 2024-09-10T19:48:47.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-13 /etc/passwd 2024-09-10T19:48:47.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-14 /etc/passwd 2024-09-10T19:48:47.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-15 /etc/passwd 2024-09-10T19:48:47.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-16 /etc/passwd 2024-09-10T19:48:47.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-17 /etc/passwd 2024-09-10T19:48:47.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:47.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-18 /etc/passwd 2024-09-10T19:48:48.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-19 /etc/passwd 2024-09-10T19:48:48.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-20 /etc/passwd 2024-09-10T19:48:48.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-21 /etc/passwd 2024-09-10T19:48:48.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-22 /etc/passwd 2024-09-10T19:48:48.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-23 /etc/passwd 2024-09-10T19:48:48.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:48.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-24 /etc/passwd 2024-09-10T19:48:49.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-25 /etc/passwd 2024-09-10T19:48:49.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-26 /etc/passwd 2024-09-10T19:48:49.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-27 /etc/passwd 2024-09-10T19:48:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-28 /etc/passwd 2024-09-10T19:48:49.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-29 /etc/passwd 2024-09-10T19:48:49.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-30 /etc/passwd 2024-09-10T19:48:49.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:49.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-31 /etc/passwd 2024-09-10T19:48:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-32 /etc/passwd 2024-09-10T19:48:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-33 /etc/passwd 2024-09-10T19:48:50.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-34 /etc/passwd 2024-09-10T19:48:50.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-35 /etc/passwd 2024-09-10T19:48:50.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-36 /etc/passwd 2024-09-10T19:48:50.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-37 /etc/passwd 2024-09-10T19:48:50.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:50.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-38 /etc/passwd 2024-09-10T19:48:51.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-39 /etc/passwd 2024-09-10T19:48:51.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-40 /etc/passwd 2024-09-10T19:48:51.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-41 /etc/passwd 2024-09-10T19:48:51.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-42 /etc/passwd 2024-09-10T19:48:51.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-43 /etc/passwd 2024-09-10T19:48:51.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:51.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-44 /etc/passwd 2024-09-10T19:48:52.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-45 /etc/passwd 2024-09-10T19:48:52.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-46 /etc/passwd 2024-09-10T19:48:52.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-47 /etc/passwd 2024-09-10T19:48:52.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-48 /etc/passwd 2024-09-10T19:48:52.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-49 /etc/passwd 2024-09-10T19:48:52.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-50 /etc/passwd 2024-09-10T19:48:52.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:52.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-51 /etc/passwd 2024-09-10T19:48:53.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.102 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-52 /etc/passwd 2024-09-10T19:48:53.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-53 /etc/passwd 2024-09-10T19:48:53.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-54 /etc/passwd 2024-09-10T19:48:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-55 /etc/passwd 2024-09-10T19:48:53.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-56 /etc/passwd 2024-09-10T19:48:53.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:53.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-57 /etc/passwd 2024-09-10T19:48:54.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-58 /etc/passwd 2024-09-10T19:48:54.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-59 /etc/passwd 2024-09-10T19:48:54.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-60 /etc/passwd 2024-09-10T19:48:54.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-61 /etc/passwd 2024-09-10T19:48:54.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-62 /etc/passwd 2024-09-10T19:48:54.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-63 /etc/passwd 2024-09-10T19:48:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-64 /etc/passwd 2024-09-10T19:48:55.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-65 /etc/passwd 2024-09-10T19:48:55.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-66 /etc/passwd 2024-09-10T19:48:55.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-67 /etc/passwd 2024-09-10T19:48:55.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-68 /etc/passwd 2024-09-10T19:48:55.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-69 /etc/passwd 2024-09-10T19:48:55.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:55.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-70 /etc/passwd 2024-09-10T19:48:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-71 /etc/passwd 2024-09-10T19:48:56.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-72 /etc/passwd 2024-09-10T19:48:56.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-73 /etc/passwd 2024-09-10T19:48:56.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-74 /etc/passwd 2024-09-10T19:48:56.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-75 /etc/passwd 2024-09-10T19:48:56.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-76 /etc/passwd 2024-09-10T19:48:56.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:56.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-77 /etc/passwd 2024-09-10T19:48:57.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-78 /etc/passwd 2024-09-10T19:48:57.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-79 /etc/passwd 2024-09-10T19:48:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-80 /etc/passwd 2024-09-10T19:48:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-81 /etc/passwd 2024-09-10T19:48:57.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-82 /etc/passwd 2024-09-10T19:48:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:57.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-83 /etc/passwd 2024-09-10T19:48:58.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-84 /etc/passwd 2024-09-10T19:48:58.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-85 /etc/passwd 2024-09-10T19:48:58.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-86 /etc/passwd 2024-09-10T19:48:58.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-87 /etc/passwd 2024-09-10T19:48:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-88 /etc/passwd 2024-09-10T19:48:58.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-89 /etc/passwd 2024-09-10T19:48:58.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:58.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-90 /etc/passwd 2024-09-10T19:48:59.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-91 /etc/passwd 2024-09-10T19:48:59.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-92 /etc/passwd 2024-09-10T19:48:59.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-93 /etc/passwd 2024-09-10T19:48:59.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-94 /etc/passwd 2024-09-10T19:48:59.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-95 /etc/passwd 2024-09-10T19:48:59.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:48:59.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-96 /etc/passwd 2024-09-10T19:49:00.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-97 /etc/passwd 2024-09-10T19:49:00.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-98 /etc/passwd 2024-09-10T19:49:00.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-99 /etc/passwd 2024-09-10T19:49:00.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-100 /etc/passwd 2024-09-10T19:49:00.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-101 /etc/passwd 2024-09-10T19:49:00.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:00.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-102 /etc/passwd 2024-09-10T19:49:01.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-103 /etc/passwd 2024-09-10T19:49:01.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-104 /etc/passwd 2024-09-10T19:49:01.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-105 /etc/passwd 2024-09-10T19:49:01.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-106 /etc/passwd 2024-09-10T19:49:01.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-107 /etc/passwd 2024-09-10T19:49:01.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-108 /etc/passwd 2024-09-10T19:49:01.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:01.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-109 /etc/passwd 2024-09-10T19:49:02.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-110 /etc/passwd 2024-09-10T19:49:02.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-111 /etc/passwd 2024-09-10T19:49:02.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-112 /etc/passwd 2024-09-10T19:49:02.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-113 /etc/passwd 2024-09-10T19:49:02.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-114 /etc/passwd 2024-09-10T19:49:02.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:02.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-115 /etc/passwd 2024-09-10T19:49:03.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-116 /etc/passwd 2024-09-10T19:49:03.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-117 /etc/passwd 2024-09-10T19:49:03.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-118 /etc/passwd 2024-09-10T19:49:03.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-119 /etc/passwd 2024-09-10T19:49:03.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-120 /etc/passwd 2024-09-10T19:49:03.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-121 /etc/passwd 2024-09-10T19:49:03.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:03.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-122 /etc/passwd 2024-09-10T19:49:04.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-123 /etc/passwd 2024-09-10T19:49:04.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-124 /etc/passwd 2024-09-10T19:49:04.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-125 /etc/passwd 2024-09-10T19:49:04.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-126 /etc/passwd 2024-09-10T19:49:04.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-127 /etc/passwd 2024-09-10T19:49:04.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:04.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-128 /etc/passwd 2024-09-10T19:49:05.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-129 /etc/passwd 2024-09-10T19:49:05.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-130 /etc/passwd 2024-09-10T19:49:05.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-131 /etc/passwd 2024-09-10T19:49:05.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-132 /etc/passwd 2024-09-10T19:49:05.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-133 /etc/passwd 2024-09-10T19:49:05.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:05.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-134 /etc/passwd 2024-09-10T19:49:06.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-135 /etc/passwd 2024-09-10T19:49:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-136 /etc/passwd 2024-09-10T19:49:06.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-137 /etc/passwd 2024-09-10T19:49:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-138 /etc/passwd 2024-09-10T19:49:06.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-139 /etc/passwd 2024-09-10T19:49:06.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:06.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-140 /etc/passwd 2024-09-10T19:49:07.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-141 /etc/passwd 2024-09-10T19:49:07.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-142 /etc/passwd 2024-09-10T19:49:07.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-143 /etc/passwd 2024-09-10T19:49:07.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-144 /etc/passwd 2024-09-10T19:49:07.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-145 /etc/passwd 2024-09-10T19:49:07.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-146 /etc/passwd 2024-09-10T19:49:07.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:07.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-147 /etc/passwd 2024-09-10T19:49:08.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:08.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-148 /etc/passwd 2024-09-10T19:49:08.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:08.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-149 /etc/passwd 2024-09-10T19:49:08.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:49:08.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-150 /etc/passwd 2024-09-10T19:49:08.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:49:08.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: jq '.pg_stats[0].up[]' 2024-09-10T19:49:08.946 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:49:08.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph osd out 1 0 2 2024-09-10T19:49:09.555 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2024-09-10T19:49:09.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:69: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:49:09.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: sleep 1 2024-09-10T19:49:10.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:76: _common_test: wait_for_clean 2024-09-10T19:49:10.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:49:10.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:49:10.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:49:10.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:49:10.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:49:10.569 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:49:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:49:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:49:10.569 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:49:10.665 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:49:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:49:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:49:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.166 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:49:11.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836524 2024-09-10T19:49:11.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836524 2024-09-10T19:49:11.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524' 2024-09-10T19:49:11.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.325 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:49:11.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672998 2024-09-10T19:49:11.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672998 2024-09-10T19:49:11.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672998' 2024-09-10T19:49:11.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.503 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:49:11.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509470 2024-09-10T19:49:11.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509470 2024-09-10T19:49:11.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509470' 2024-09-10T19:49:11.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:49:11.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345948 2024-09-10T19:49:11.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345948 2024-09-10T19:49:11.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509470 3-85899345948' 2024-09-10T19:49:11.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:49:11.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182418 2024-09-10T19:49:11.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182418 2024-09-10T19:49:11.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509470 3-85899345948 4-107374182418' 2024-09-10T19:49:11.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:11.918 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018890 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018890 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509470 3-85899345948 4-107374182418 5-128849018890' 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836524 2024-09-10T19:49:12.033 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:12.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:49:12.034 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836524 2024-09-10T19:49:12.035 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:12.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836524 2024-09-10T19:49:12.036 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836524 2024-09-10T19:49:12.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-10T19:49:12.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:49:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-10T19:49:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:12.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672998 2024-09-10T19:49:12.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:12.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:49:12.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672998 2024-09-10T19:49:12.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:12.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672998 2024-09-10T19:49:12.426 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672998 2024-09-10T19:49:12.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-09-10T19:49:12.426 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:49:12.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672999 -lt 42949672998 2024-09-10T19:49:12.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:12.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509470 2024-09-10T19:49:12.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:12.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:49:12.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509470 2024-09-10T19:49:12.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:12.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509470 2024-09-10T19:49:12.971 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509470 2024-09-10T19:49:12.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509470' 2024-09-10T19:49:12.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:49:13.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509470 -lt 64424509470 2024-09-10T19:49:13.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:13.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345948 2024-09-10T19:49:13.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:49:13.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345948 2024-09-10T19:49:13.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345948 2024-09-10T19:49:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345948' 2024-09-10T19:49:13.525 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345948 2024-09-10T19:49:13.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:49:14.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345948 -lt 85899345948 2024-09-10T19:49:14.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:14.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182418 2024-09-10T19:49:14.094 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:14.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:49:14.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182418 2024-09-10T19:49:14.095 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:14.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182418 2024-09-10T19:49:14.096 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182418 2024-09-10T19:49:14.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182418' 2024-09-10T19:49:14.097 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:49:14.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182418 -lt 107374182418 2024-09-10T19:49:14.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:14.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018890 2024-09-10T19:49:14.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:14.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:49:14.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018890 2024-09-10T19:49:14.757 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:14.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018890 2024-09-10T19:49:14.758 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018890 2024-09-10T19:49:14.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-09-10T19:49:14.758 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:49:15.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018891 -lt 128849018890 2024-09-10T19:49:15.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:49:15.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:15.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:15.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T19:49:15.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:15.994 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:15.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:15.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:15.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:15.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:15.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:16.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:16.375 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:16.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:16.376 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:17.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:17.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T19:49:17.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:49:17.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T19:49:17.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:17.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:17.124 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:17.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:17.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:17.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:17.575 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:18.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:18.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:18.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:18.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:18.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:18.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:18.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:18.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:18.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:18.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=13605 2024-09-10T19:49:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 13605 '!=' null 2024-09-10T19:49:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:18.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:18.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:18.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:18.944 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:19.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:19.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:19.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:19.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:20.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:20.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:20.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=49219 2024-09-10T19:49:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 49219 '!=' null 2024-09-10T19:49:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:20.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:20.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:20.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:20.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:20.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:21.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:21.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:21.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:21.170 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:21.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:21.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:21.771 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:22.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=49219 2024-09-10T19:49:22.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 49219 '!=' null 2024-09-10T19:49:22.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:22.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:22.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:22.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:22.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:22.541 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:23.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:23.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:23.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:23.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:23.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:23.581 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=42195 2024-09-10T19:49:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 42195 '!=' null 2024-09-10T19:49:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:24.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:24.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:24.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:24.471 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:24.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:24.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:24.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:24.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:24.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:24.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:24.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:24.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:24.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:25.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:25.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:25.493 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:26.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=66010 2024-09-10T19:49:26.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 66010 '!=' null 2024-09-10T19:49:26.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:26.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:26.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:26.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:26.282 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:26.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:26.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:26.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:26.817 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:27.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:27.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:27.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:27.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:27.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:27.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:27.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:27.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:27.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:28.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57492 2024-09-10T19:49:28.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57492 '!=' null 2024-09-10T19:49:28.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:28.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:28.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:28.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:28.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:28.146 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:28.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:28.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:28.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:28.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:28.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:28.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:28.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:28.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:28.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:29.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:29.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:29.312 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:29.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57491 2024-09-10T19:49:29.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57491 '!=' null 2024-09-10T19:49:29.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:29.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:29.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:30.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:30.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:30.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:30.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:30.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:30.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:30.488 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:31.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:31.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:31.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:31.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:31.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:31.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:31.147 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=87302 2024-09-10T19:49:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 87302 '!=' null 2024-09-10T19:49:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:31.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:31.887 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:31.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:31.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:31.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:31.888 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:31.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:31.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:32.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:32.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:32.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:32.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:32.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:32.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:32.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:32.954 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:33.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57491 2024-09-10T19:49:33.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57491 '!=' null 2024-09-10T19:49:33.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:33.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:33.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:33.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:33.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:33.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:34.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:34.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:34.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:34.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:34.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:34.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:34.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:34.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:34.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:34.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:34.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:34.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:34.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:35.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=85172 2024-09-10T19:49:35.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 85172 '!=' null 2024-09-10T19:49:35.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:35.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:35.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:35.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:35.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:35.329 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:35.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:35.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:35.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:35.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:36.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:36.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:36.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:36.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:37.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57491 2024-09-10T19:49:37.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57491 '!=' null 2024-09-10T19:49:37.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:37.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:37.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:37.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:37.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:37.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:37.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:37.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:37.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:37.898 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:38.428 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:39.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57491 2024-09-10T19:49:39.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57491 '!=' null 2024-09-10T19:49:39.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:39.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:39.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:39.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:39.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:39.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:39.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:39.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:39.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:39.727 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:39.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:39.728 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:40.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:40.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:40.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:40.293 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:40.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=87303 2024-09-10T19:49:40.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 87303 '!=' null 2024-09-10T19:49:40.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:40.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:40.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:41.023 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:41.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:41.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:41.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:41.403 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:42.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:42.045 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:42.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=44717 2024-09-10T19:49:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 44717 '!=' null 2024-09-10T19:49:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:42.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:42.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:42.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:42.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:42.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:42.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:42.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:42.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:42.769 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:43.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:43.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:43.144 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:43.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:43.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:43.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:43.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:43.669 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:44.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=44717 2024-09-10T19:49:44.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 44717 '!=' null 2024-09-10T19:49:44.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:44.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:44.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:44.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:44.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:44.320 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:44.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:44.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:44.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:44.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:44.321 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:44.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:49:44.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:44.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:44.697 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:45.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:49:45.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:49:45.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:49:45.222 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:49:45.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=44716 2024-09-10T19:49:45.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 44716 '!=' null 2024-09-10T19:49:45.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:49:45.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:49:45.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:49:45.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:49:46.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:49:46.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:49:46.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:49:46.253 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:49:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T19:49:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:49:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:49:46.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:78: _common_test: flush_pg_stats 2024-09-10T19:49:46.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:49:46.811 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:49:47.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:49:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836533 2024-09-10T19:49:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836533 2024-09-10T19:49:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533' 2024-09-10T19:49:47.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:49:47.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673008 2024-09-10T19:49:47.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673008 2024-09-10T19:49:47.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673008' 2024-09-10T19:49:47.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.460 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:49:47.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509479 2024-09-10T19:49:47.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509479 2024-09-10T19:49:47.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673008 2-64424509479' 2024-09-10T19:49:47.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.576 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:49:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345957 2024-09-10T19:49:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345957 2024-09-10T19:49:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673008 2-64424509479 3-85899345957' 2024-09-10T19:49:47.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:49:47.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182427 2024-09-10T19:49:47.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182427 2024-09-10T19:49:47.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673008 2-64424509479 3-85899345957 4-107374182427' 2024-09-10T19:49:47.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:49:47.806 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:49:47.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018900 2024-09-10T19:49:47.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018900 2024-09-10T19:49:47.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836533 1-42949673008 2-64424509479 3-85899345957 4-107374182427 5-128849018900' 2024-09-10T19:49:47.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:47.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836533 2024-09-10T19:49:47.922 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:47.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:49:47.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836533 2024-09-10T19:49:47.923 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:47.924 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836533 2024-09-10T19:49:47.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836533 2024-09-10T19:49:47.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-09-10T19:49:47.925 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:49:48.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836533 -lt 21474836533 2024-09-10T19:49:48.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:48.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673008 2024-09-10T19:49:48.358 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:48.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:49:48.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673008 2024-09-10T19:49:48.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:48.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673008 2024-09-10T19:49:48.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-10T19:49:48.361 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673008 2024-09-10T19:49:48.361 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:49:48.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-10T19:49:48.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:48.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509479 2024-09-10T19:49:48.784 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:48.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:49:48.785 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509479 2024-09-10T19:49:48.786 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:48.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509479 2024-09-10T19:49:48.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-10T19:49:48.787 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509479 2024-09-10T19:49:48.787 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:49:49.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509479 -lt 64424509479 2024-09-10T19:49:49.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:49.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345957 2024-09-10T19:49:49.210 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:49.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:49:49.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345957 2024-09-10T19:49:49.211 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:49.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345957 2024-09-10T19:49:49.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2024-09-10T19:49:49.213 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345957 2024-09-10T19:49:49.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:49:49.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345957 -lt 85899345957 2024-09-10T19:49:49.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:49.636 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182427 2024-09-10T19:49:49.637 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:49.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:49:49.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182427 2024-09-10T19:49:49.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:49.639 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182427 2024-09-10T19:49:49.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182427 2024-09-10T19:49:49.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-09-10T19:49:49.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:49:50.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182427 -lt 107374182427 2024-09-10T19:49:50.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:49:50.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018900 2024-09-10T19:49:50.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:49:50.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:49:50.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018900 2024-09-10T19:49:50.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:49:50.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018900 2024-09-10T19:49:50.065 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018900 2024-09-10T19:49:50.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-09-10T19:49:50.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:49:50.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018900 -lt 128849018900 2024-09-10T19:49:50.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:49:50.487 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: jq '.pg_stats[0].up_primary' 2024-09-10T19:49:50.854 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:49:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: newprimary=4 2024-09-10T19:49:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:81: _common_test: kill_daemons 2024-09-10T19:49:50.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:49:50.866 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:49:50.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:49:50.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:49:50.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:49:56.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:49:56.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: ERRORS=0 2024-09-10T19:49:56.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: tee td/osd-backfill-recovery-log/result.log 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=4 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:49:56.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:49:57.777 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:49:57.777 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2024-09-10T19:49:57.777 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "34'150", 2024-09-10T19:49:57.777 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "34'148", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e9c179f9:::obj-149:head", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'149", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4817.0:1", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:49:08.295780+0000", 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:49:57.778 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.779 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cb39fd60:::obj-150:head", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'150", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4820.0:1", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:49:08.449241+0000", 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:49:57.780 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:49:57.781 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:49:57.781 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.781 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.781 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.782 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [ 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4793.0:1", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'141", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "141", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4796.0:1", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'142", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "142", 2024-09-10T19:49:57.783 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4799.0:1", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'143", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "143", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4802.0:1", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'144", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "144", 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.784 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4805.0:1", 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'145", 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "145", 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4808.0:1", 2024-09-10T19:49:57.785 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'146", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "146", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4811.0:1", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'147", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "147", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4814.0:1", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'148", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "148", 2024-09-10T19:49:57.786 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:49:57.787 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:49:57.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:49:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: LOGLEN=2 2024-09-10T19:49:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:86: _common_test: '[' 2 '!=' 2 ']' 2024-09-10T19:49:57.788 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:49:57.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: DUPSLEN=8 2024-09-10T19:49:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:91: _common_test: '[' 8 '!=' 8 ']' 2024-09-10T19:49:57.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=7 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'150 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'149 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to END max 2 dups.size()=8 other.dups.size()=7 2024-09-10T19:49:57.790 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=7 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'150 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'149 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to END max 2 dups.size()=8 other.dups.size()=7 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=7 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'150 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to copy log version 34'149 2024-09-10T19:49:57.791 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:49:57.792 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:49:57.792 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:49:57.820 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:49:57.820 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:49:57.820 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:11.310+0000 7fe01a0e9640 20 copy_up_to END max 2 dups.size()=8 other.dups.size()=7 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_after v 34'150 dups.size()=0 other.dups.size()=7 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:49:57.821 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'149 2024-09-10T19:49:57.822 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_up_to/copy_after copy dup from log version 34'150 2024-09-10T19:49:57.822 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:49:40.520+0000 7fe01b0eb640 20 copy_after END v 34'150 dups.size()=10 other.dups.size()=7 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:97: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:35: run: teardown td/osd-backfill-recovery-log 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:49:57.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:49:57.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:49:57.830 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:49:57.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:49:57.831 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:49:57.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:49:57.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:49:57.832 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:49:57.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:49:57.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:49:57.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:49:57.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:49:57.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:49:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:49:57.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:49:57.873 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:49:57.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:57.874 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:57.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:49:57.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: for func in $funcs 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: setup td/osd-backfill-recovery-log 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-recovery-log 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:49:57.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:49:57.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:49:57.877 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:49:57.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:49:57.878 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:49:57.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:49:57.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:49:57.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:49:57.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:49:57.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:49:57.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:49:57.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:49:57.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:49:57.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:49:57.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:49:57.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:49:57.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:57.884 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:57.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:49:57.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:49:57.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:49:57.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-recovery-log 2024-09-10T19:49:57.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:49:57.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:57.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:57.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53443 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:114: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:116: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local loglen=2 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local dupslen=148 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local objects=150 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:46: _common_test: local moreobjects=0 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:48: _common_test: local OSDS=6 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mon td/osd-backfill-recovery-log a 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-recovery-log/a 2024-09-10T19:49:57.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:58.363 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:49:58.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:49:58.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:49:58.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:49:58.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:49:58.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:49:58.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:49:58.565 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:49:58.566 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get fsid 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:49:58.648 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:58.649 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:58.649 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:49:58.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:49:58.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get mon_host 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: run_mgr td/osd-backfill-recovery-log x 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-recovery-log/x 2024-09-10T19:49:58.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:49:59.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:49:59.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:49:59.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export CEPH_ARGS 2024-09-10T19:49:59.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:49:59.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:49:59.610 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: expr 6 - 1 2024-09-10T19:49:59.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: seq 0 5 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 0 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:49:59.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:49:59.616 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:49:59.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3' 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3 2024-09-10T19:49:59.618 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:49:59.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDnouBmCxzCJRAA+6T4gZp6Q47xO0CDt750NQ== 2024-09-10T19:49:59.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDnouBmCxzCJRAA+6T4gZp6Q47xO0CDt750NQ=="}' 2024-09-10T19:49:59.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3 -i td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:49:59.905 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:49:59.914 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:49:59.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnouBmCxzCJRAA+6T4gZp6Q47xO0CDt750NQ== --osd-uuid f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3 2024-09-10T19:49:59.949 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:49:59.945+0000 7feef81a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:49:59.973 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:49:59.970+0000 7feef81a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:49:59.974 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:49:59.971+0000 7feef81a2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:49:59.975 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:49:59.972+0000 7feef81a2640 -1 bdev(0x563abf295c00 td/osd-backfill-recovery-log/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:49:59.975 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:49:59.972+0000 7feef81a2640 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2024-09-10T19:50:05.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2024-09-10T19:50:05.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:50:05.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:50:05.431 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:50:05.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:50:05.973 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:50:05.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:50:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:50:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:50:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:50:05.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:50:06.004 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:06.001+0000 7f2aafc10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:06.006 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:06.003+0000 7f2aafc10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:06.007 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:06.004+0000 7f2aafc10640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:06.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:06.398 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:50:06.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:50:06.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:06.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:06.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:07.353 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:07.350+0000 7f2aafc10640 -1 Falling back to public interface 2024-09-10T19:50:07.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:07.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:07.824 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:50:07.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:50:07.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:07.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:08.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:08.944 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:08.942+0000 7f2aafc10640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:50:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:09.251 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:50:09.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:50:09.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:09.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:09.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:10.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:10.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:10.668 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:50:10.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:50:10.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:10.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:11.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:12.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:12.063 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:50:12.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:12.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:50:12.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:12.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:12.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:13.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:13.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:13.483 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:50:13.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:50:13.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:13.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:13.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:14.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:14.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:14.966 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:50:14.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:50:14.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:14.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:15.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:16.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:16.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:16.386 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:50:16.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:50:16.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:16.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:16.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:17.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:17.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:17.794 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:50:17.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:50:17.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:17.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:18.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:19.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:19.196 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:50:19.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:50:19.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:19.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:19.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:20.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:20.645 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:50:20.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:50:20.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:20.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:21.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:22.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:22.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:22.506 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:50:22.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:50:22.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:22.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:22.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:23.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:23.992 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:50:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:50:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:23.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:24.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:25.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:25.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:25.889 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:50:25.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:50:25.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:25.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:26.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:27.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:27.775 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:50:27.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:50:27.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:27.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:28.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:29.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:29.549 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:50:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:50:29.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:29.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:30.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:31.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:31.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:31.737 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:50:31.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:50:31.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:31.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:50:32.126 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3407302154,v1:127.0.0.1:6803/3407302154] [v2:127.0.0.1:6804/3407302154,v1:127.0.0.1:6805/3407302154] exists,up f6bf594d-8d00-4c1b-a3b8-e71ccdd4a2f3 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 1 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:50:32.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:50:32.131 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:50:32.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f6f67146-6fbe-4dfc-9287-bcf0f6f5550b 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 f6f67146-6fbe-4dfc-9287-bcf0f6f5550b 2024-09-10T19:50:32.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 f6f67146-6fbe-4dfc-9287-bcf0f6f5550b' 2024-09-10T19:50:32.134 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:50:32.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAIo+BmOQ7NCBAAh6VzAdd9RIWaLPfOCbKEnQ== 2024-09-10T19:50:32.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAIo+BmOQ7NCBAAh6VzAdd9RIWaLPfOCbKEnQ=="}' 2024-09-10T19:50:32.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f6f67146-6fbe-4dfc-9287-bcf0f6f5550b -i td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:50:32.621 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:50:32.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:50:32.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAIo+BmOQ7NCBAAh6VzAdd9RIWaLPfOCbKEnQ== --osd-uuid f6f67146-6fbe-4dfc-9287-bcf0f6f5550b 2024-09-10T19:50:32.667 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:32.665+0000 7f5ce3215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:32.681 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:32.679+0000 7f5ce3215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:32.682 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:32.680+0000 7f5ce3215640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:32.683 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:32.680+0000 7f5ce3215640 -1 bdev(0x5612f3be5c00 td/osd-backfill-recovery-log/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:50:32.683 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:32.680+0000 7f5ce3215640 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2024-09-10T19:50:38.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2024-09-10T19:50:38.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:50:38.683 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:50:38.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:50:38.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:50:39.218 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:50:39.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:50:39.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:50:39.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:50:39.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:50:39.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:50:39.249 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:39.247+0000 7fc053c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:39.265 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:39.263+0000 7fc053c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:39.266 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:39.264+0000 7fc053c19640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:50:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:50:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:50:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:50:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:50:39.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:50:39.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:39.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:50:39.641 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:50:39.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:39.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:40.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:40.361 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:40.359+0000 7fc053c19640 -1 Falling back to public interface 2024-09-10T19:50:41.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:41.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:41.073 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:50:41.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:50:41.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:41.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:41.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:42.210 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:50:42.208+0000 7fc053c19640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:50:42.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:42.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:42.550 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:50:42.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:50:42.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:42.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:42.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:43.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:43.987 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:43.988 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:50:43.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:50:43.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:43.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:44.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:45.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:45.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:45.393 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:50:45.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:50:45.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:45.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:45.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:46.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:46.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:46.798 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:50:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:50:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:46.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:47.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:48.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:48.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:48.199 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:50:48.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:50:48.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:48.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:48.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:49.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:49.604 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:50:49.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:50:49.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:49.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:50.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:51.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:51.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:51.085 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:50:51.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:50:51.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:51.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:51.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:52.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:52.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:52.483 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:50:52.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:50:52.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:52.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:53.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:54.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:54.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:54.050 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:50:54.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:50:54.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:54.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:54.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:55.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:55.447 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:50:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:50:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:55.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:56.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:57.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:57.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:57.035 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:50:57.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:50:57.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:57.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:57.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:50:58.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:50:58.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:50:58.650 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:50:58.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:50:58.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:50:58.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:50:59.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:00.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:00.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:00.567 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:51:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:51:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:00.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:51:01.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:02.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:02.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:02.321 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:51:02.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:51:02.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:02.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:51:02.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:03.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:03.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:03.957 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:51:03.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:51:03.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:03.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:51:04.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:05.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:05.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:05.813 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:51:05.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:51:05.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:05.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:51:06.264 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1992508315,v1:127.0.0.1:6811/1992508315] [v2:127.0.0.1:6812/1992508315,v1:127.0.0.1:6813/1992508315] exists,up f6f67146-6fbe-4dfc-9287-bcf0f6f5550b 2024-09-10T19:51:06.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 2 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:51:06.268 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:51:06.269 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:51:06.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=67f23773-bcdc-409d-9db2-38d5bbec0a3e 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 67f23773-bcdc-409d-9db2-38d5bbec0a3e 2024-09-10T19:51:06.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 67f23773-bcdc-409d-9db2-38d5bbec0a3e' 2024-09-10T19:51:06.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:51:06.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqo+Bma18LERAA3kAQWJvqmZ8ao6buEvP4TQ== 2024-09-10T19:51:06.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqo+Bma18LERAA3kAQWJvqmZ8ao6buEvP4TQ=="}' 2024-09-10T19:51:06.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 67f23773-bcdc-409d-9db2-38d5bbec0a3e -i td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:51:06.784 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:51:06.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:51:06.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAqo+Bma18LERAA3kAQWJvqmZ8ao6buEvP4TQ== --osd-uuid 67f23773-bcdc-409d-9db2-38d5bbec0a3e 2024-09-10T19:51:06.829 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:06.827+0000 7f4885413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:06.833 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:06.831+0000 7f4885413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:06.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:06.832+0000 7f4885413640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:06.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:06.832+0000 7f4885413640 -1 bdev(0x55bc2c164800 td/osd-backfill-recovery-log/2/block) open stat got: (1) Operation not permitted 2024-09-10T19:51:06.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:06.832+0000 7f4885413640 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2024-09-10T19:51:12.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2024-09-10T19:51:12.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:51:12.938 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T19:51:12.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T19:51:12.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:51:13.477 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T19:51:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T19:51:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:51:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:51:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:51:13.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:51:13.510 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:13.507+0000 7fc55da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:13.520 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:13.518+0000 7fc55da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:13.521 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:13.519+0000 7fc55da0b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:13.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T19:51:13.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:51:13.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:13.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:14.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:14.610 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:14.608+0000 7fc55da0b640 -1 Falling back to public interface 2024-09-10T19:51:15.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:15.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:15.387 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:51:15.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:51:15.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:15.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:15.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:16.340 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:16.338+0000 7fc55da0b640 -1 osd.2 0 log_to_monitors true 2024-09-10T19:51:16.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:16.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:16.839 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:51:16.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:51:16.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:16.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:17.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:18.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:18.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:18.400 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:51:18.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:51:18.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:18.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:18.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:19.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:19.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:19.825 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:51:19.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:51:19.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:19.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:20.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:21.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:21.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:21.416 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:51:21.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:51:21.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:21.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:21.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:22.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:22.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:22.814 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:51:22.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:51:22.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:22.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:23.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:24.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:24.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:24.283 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:51:24.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:51:24.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:24.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:24.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:25.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:25.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:25.686 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:51:25.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:51:25.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:25.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:26.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:27.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:27.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:27.090 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:51:27.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:51:27.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:27.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:27.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:28.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:28.497 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:51:28.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:51:28.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:28.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:29.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:30.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:30.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:30.086 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:51:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:51:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:30.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:31.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:32.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:32.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:32.199 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:51:32.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:51:32.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:32.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:33.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:34.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:34.011 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:51:34.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:51:34.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:34.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:34.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:35.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:35.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:35.739 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:51:35.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:51:35.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:35.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:36.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:37.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:37.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:37.648 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:51:37.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:51:37.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:37.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:38.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:39.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:39.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:39.312 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:51:39.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:51:39.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:39.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:39.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:40.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:40.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:40.797 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:51:40.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:51:40.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:40.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:51:41.234 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1136616726,v1:127.0.0.1:6819/1136616726] [v2:127.0.0.1:6820/1136616726,v1:127.0.0.1:6821/1136616726] exists,up 67f23773-bcdc-409d-9db2-38d5bbec0a3e 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 3 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2024-09-10T19:51:41.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:51:41.239 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:51:41.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=812437c4-1542-475c-adad-4eac8919165d 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 812437c4-1542-475c-adad-4eac8919165d' 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 812437c4-1542-475c-adad-4eac8919165d 2024-09-10T19:51:41.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBNo+BmCyA5DxAAqdr9Py+UtJN0PqaqeM/HRw== 2024-09-10T19:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBNo+BmCyA5DxAAqdr9Py+UtJN0PqaqeM/HRw=="}' 2024-09-10T19:51:41.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 812437c4-1542-475c-adad-4eac8919165d -i td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:51:41.764 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:51:41.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:51:41.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNo+BmCyA5DxAAqdr9Py+UtJN0PqaqeM/HRw== --osd-uuid 812437c4-1542-475c-adad-4eac8919165d 2024-09-10T19:51:41.810 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:41.808+0000 7fa30ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:41.812 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:41.810+0000 7fa30ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:41.813 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:41.811+0000 7fa30ee16640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:41.813 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:41.811+0000 7fa30ee16640 -1 bdev(0x56406448dc00 td/osd-backfill-recovery-log/3/block) open stat got: (1) Operation not permitted 2024-09-10T19:51:41.813 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:41.811+0000 7fa30ee16640 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2024-09-10T19:51:47.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2024-09-10T19:51:47.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:51:47.272 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-10T19:51:47.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T19:51:47.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:51:47.815 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-10T19:51:47.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T19:51:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:51:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:51:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:51:47.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:51:47.846 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:47.843+0000 7fb298c42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:47.848 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:47.845+0000 7fb298c42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:47.849 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:47.846+0000 7fb298c42640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:51:48.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T19:51:48.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:51:48.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:48.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:48.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:48.941 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:48.938+0000 7fb298c42640 -1 Falling back to public interface 2024-09-10T19:51:49.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:49.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:49.662 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:51:49.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:51:49.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:49.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:50.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:50.593 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:51:50.587+0000 7fb298c42640 -1 osd.3 0 log_to_monitors true 2024-09-10T19:51:51.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:51.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:51.179 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:51:51.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:51:51.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:51.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:51.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:52.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:52.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:52.631 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:51:52.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:51:52.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:52.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:53.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:54.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:54.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:54.061 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:51:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:51:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:54.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:54.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:55.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:55.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:51:55.554 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:51:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:55.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:55.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:56.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:56.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:56.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:51:56.984 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:51:56.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:56.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:57.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:58.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:58.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:58.406 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:51:58.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:51:58.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:58.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:51:58.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:51:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:51:59.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:51:59.807 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:51:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:51:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:51:59.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:00.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:01.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:01.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:01.495 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:52:01.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:52:01.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:01.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:01.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:02.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:02.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:02.967 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:52:02.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:52:02.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:02.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:03.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:04.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:04.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:04.981 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:52:04.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:52:04.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:04.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:05.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:06.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:06.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:06.760 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:52:06.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:52:06.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:06.761 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:07.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:08.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:08.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:08.538 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:52:08.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:52:08.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:08.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:09.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:10.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:10.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:10.325 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:52:10.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:52:10.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:10.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:11.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:12.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:12.421 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:52:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:52:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:12.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:13.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:14.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:14.183 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:52:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:52:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:14.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:52:14.577 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3202751170,v1:127.0.0.1:6827/3202751170] [v2:127.0.0.1:6828/3202751170,v1:127.0.0.1:6829/3202751170] exists,up 812437c4-1542-475c-adad-4eac8919165d 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 4 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:52:14.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:52:14.583 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:52:14.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a9fd8033-8aae-4076-ad0b-24914c15271f 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 a9fd8033-8aae-4076-ad0b-24914c15271f 2024-09-10T19:52:14.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 a9fd8033-8aae-4076-ad0b-24914c15271f' 2024-09-10T19:52:14.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:52:14.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBuo+Bm0KS8IxAAyyyu6aK5V+qC4qx6PXMrqw== 2024-09-10T19:52:14.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBuo+Bm0KS8IxAAyyyu6aK5V+qC4qx6PXMrqw=="}' 2024-09-10T19:52:14.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a9fd8033-8aae-4076-ad0b-24914c15271f -i td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:52:15.068 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:52:15.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:52:15.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuo+Bm0KS8IxAAyyyu6aK5V+qC4qx6PXMrqw== --osd-uuid a9fd8033-8aae-4076-ad0b-24914c15271f 2024-09-10T19:52:15.113 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:15.110+0000 7eff13158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:15.128 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:15.126+0000 7eff13158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:15.129 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:15.127+0000 7eff13158640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:15.129 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:15.127+0000 7eff13158640 -1 bdev(0x5649e01e5c00 td/osd-backfill-recovery-log/4/block) open stat got: (1) Operation not permitted 2024-09-10T19:52:15.129 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:15.127+0000 7eff13158640 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2024-09-10T19:52:21.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2024-09-10T19:52:21.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:52:21.085 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-10T19:52:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T19:52:21.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:52:21.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T19:52:21.625 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-10T19:52:21.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:52:21.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:52:21.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:52:21.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:52:21.656 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:21.654+0000 7f8a4c415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:21.667 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:21.665+0000 7f8a4c415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:21.668 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:21.666+0000 7f8a4c415640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:22.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T19:52:22.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:52:22.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T19:52:22.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:22.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:22.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:22.761 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:22.759+0000 7f8a4c415640 -1 Falling back to public interface 2024-09-10T19:52:23.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:23.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:23.527 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:52:23.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:52:23.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:23.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:23.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:24.537 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:24.535+0000 7f8a4c415640 -1 osd.4 0 log_to_monitors true 2024-09-10T19:52:24.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:24.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:24.934 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:52:24.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:52:24.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:24.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:25.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:26.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:26.353 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:52:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:52:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:26.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:26.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:27.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:27.819 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:52:27.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:52:27.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:27.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:28.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:29.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:29.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:29.365 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:52:29.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:52:29.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:29.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:29.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:30.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:30.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:30.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:52:30.767 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:52:30.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:30.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:31.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:32.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:32.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:32.176 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:52:32.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:52:32.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:32.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:32.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:33.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:33.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:33.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:52:33.700 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:52:33.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:33.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:34.207 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:35.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:35.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:35.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:52:35.209 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:52:35.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:35.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:35.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:36.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:36.816 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:36.816 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:52:36.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:52:36.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:36.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:37.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:38.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:38.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:38.388 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:52:38.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:52:38.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:38.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:39.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:40.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:40.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:40.108 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:52:40.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:52:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:40.109 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:41.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:42.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:42.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:42.339 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:52:42.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:52:42.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:42.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:43.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:44.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:44.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:44.051 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:52:44.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:52:44.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:44.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:44.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:45.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:45.552 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:45.553 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:52:45.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:52:45.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:45.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:46.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:47.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:47.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:47.414 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:52:47.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:52:47.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:47.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:47.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:48.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:48.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:48.845 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:52:48.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:52:48.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:48.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:52:49.282 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2568022570,v1:127.0.0.1:6835/2568022570] [v2:127.0.0.1:6836/2568022570,v1:127.0.0.1:6837/2568022570] exists,up a9fd8033-8aae-4076-ad0b-24914c15271f 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 5 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:52:49.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:52:49.288 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:52:49.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ebe7a099-c5fd-49f5-a072-036ddf1bf7ce 2024-09-10T19:52:49.290 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 ebe7a099-c5fd-49f5-a072-036ddf1bf7ce 2024-09-10T19:52:49.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 ebe7a099-c5fd-49f5-a072-036ddf1bf7ce' 2024-09-10T19:52:49.291 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:52:49.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRo+BmOGckEhAA7tKmn1MGtAsdlrzRUC9jbw== 2024-09-10T19:52:49.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRo+BmOGckEhAA7tKmn1MGtAsdlrzRUC9jbw=="}' 2024-09-10T19:52:49.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ebe7a099-c5fd-49f5-a072-036ddf1bf7ce -i td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:52:49.789 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:52:49.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:52:49.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRo+BmOGckEhAA7tKmn1MGtAsdlrzRUC9jbw== --osd-uuid ebe7a099-c5fd-49f5-a072-036ddf1bf7ce 2024-09-10T19:52:49.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:49.831+0000 7f38242f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:49.857 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:49.855+0000 7f38242f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:49.858 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:49.856+0000 7f38242f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:49.859 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:49.856+0000 7f38242f1640 -1 bdev(0x561378ed5c00 td/osd-backfill-recovery-log/5/block) open stat got: (1) Operation not permitted 2024-09-10T19:52:49.859 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:49.856+0000 7f38242f1640 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2024-09-10T19:52:56.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2024-09-10T19:52:56.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:52:56.011 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-10T19:52:56.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T19:52:56.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:52:56.544 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-10T19:52:56.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T19:52:56.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:52:56.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:52:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:52:56.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:52:56.576 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:56.574+0000 7ffaa4813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:56.581 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:56.578+0000 7ffaa4813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:56.582 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:56.580+0000 7ffaa4813640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:52:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:52:56.996 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:52:56.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:56.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:52:57.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:58.173 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:58.171+0000 7ffaa4813640 -1 Falling back to public interface 2024-09-10T19:52:58.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:58.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:58.423 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:52:58.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:52:58.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:58.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:52:58.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:52:59.552 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:52:59.550+0000 7ffaa4813640 -1 osd.5 0 log_to_monitors true 2024-09-10T19:52:59.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:52:59.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:52:59.858 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:52:59.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:52:59.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:52:59.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:00.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:01.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:01.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:01.257 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:53:01.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:53:01.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:01.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:01.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:02.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:02.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:02.843 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:53:02.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:53:02.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:02.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:03.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:04.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:04.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:04.248 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:53:04.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:53:04.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:04.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:04.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:05.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:05.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:05.645 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:53:05.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:53:05.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:05.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:06.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:07.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:07.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:07.093 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:53:07.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:53:07.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:07.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:07.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:08.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:08.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:08.546 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:53:08.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:53:08.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:08.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:08.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:09.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:09.974 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:53:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:53:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:09.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:10.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:11.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:11.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:11.450 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:53:11.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:53:11.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:11.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:11.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:12.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:12.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:12.850 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:53:12.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:53:12.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:12.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:13.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:14.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:14.616 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:53:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:53:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:14.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:15.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:16.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:16.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:16.435 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:53:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:53:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:16.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:17.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:18.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:18.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:18.235 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:53:18.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:53:18.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:18.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:19.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:20.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:20.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:20.058 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:53:20.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:53:20.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:20.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:21.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:22.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:22.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:22.142 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:53:22.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:53:22.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:22.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:22.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:53:23.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:53:23.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:53:23.651 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:53:23.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:53:23.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:53:23.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:53:24.074 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2887048968,v1:127.0.0.1:6843/2887048968] [v2:127.0.0.1:6844/2887048968,v1:127.0.0.1:6845/2887048968] exists,up ebe7a099-c5fd-49f5-a072-036ddf1bf7ce 2024-09-10T19:53:24.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:53:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:53:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:53:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:60: _common_test: create_pool test 1 1 2024-09-10T19:53:24.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T19:53:24.579 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2024-09-10T19:53:24.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:53:25.592 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: seq 1 150 2024-09-10T19:53:25.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:25.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-1 /etc/passwd 2024-09-10T19:53:26.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-2 /etc/passwd 2024-09-10T19:53:26.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-3 /etc/passwd 2024-09-10T19:53:26.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-4 /etc/passwd 2024-09-10T19:53:26.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-5 /etc/passwd 2024-09-10T19:53:26.739 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-6 /etc/passwd 2024-09-10T19:53:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:26.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-7 /etc/passwd 2024-09-10T19:53:27.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-8 /etc/passwd 2024-09-10T19:53:27.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-9 /etc/passwd 2024-09-10T19:53:27.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-10 /etc/passwd 2024-09-10T19:53:27.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.477 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-11 /etc/passwd 2024-09-10T19:53:27.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-12 /etc/passwd 2024-09-10T19:53:27.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-13 /etc/passwd 2024-09-10T19:53:27.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:27.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-14 /etc/passwd 2024-09-10T19:53:28.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-15 /etc/passwd 2024-09-10T19:53:28.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-16 /etc/passwd 2024-09-10T19:53:28.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-17 /etc/passwd 2024-09-10T19:53:28.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-18 /etc/passwd 2024-09-10T19:53:28.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.718 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-19 /etc/passwd 2024-09-10T19:53:28.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:28.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-20 /etc/passwd 2024-09-10T19:53:29.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-21 /etc/passwd 2024-09-10T19:53:29.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-22 /etc/passwd 2024-09-10T19:53:29.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-23 /etc/passwd 2024-09-10T19:53:29.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-24 /etc/passwd 2024-09-10T19:53:29.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-25 /etc/passwd 2024-09-10T19:53:29.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-26 /etc/passwd 2024-09-10T19:53:29.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:29.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-27 /etc/passwd 2024-09-10T19:53:30.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-28 /etc/passwd 2024-09-10T19:53:30.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-29 /etc/passwd 2024-09-10T19:53:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-30 /etc/passwd 2024-09-10T19:53:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-31 /etc/passwd 2024-09-10T19:53:30.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-32 /etc/passwd 2024-09-10T19:53:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:30.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-33 /etc/passwd 2024-09-10T19:53:31.000 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-34 /etc/passwd 2024-09-10T19:53:31.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-35 /etc/passwd 2024-09-10T19:53:31.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-36 /etc/passwd 2024-09-10T19:53:31.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-37 /etc/passwd 2024-09-10T19:53:31.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-38 /etc/passwd 2024-09-10T19:53:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-39 /etc/passwd 2024-09-10T19:53:31.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:31.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-40 /etc/passwd 2024-09-10T19:53:32.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-41 /etc/passwd 2024-09-10T19:53:32.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-42 /etc/passwd 2024-09-10T19:53:32.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-43 /etc/passwd 2024-09-10T19:53:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-44 /etc/passwd 2024-09-10T19:53:32.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-45 /etc/passwd 2024-09-10T19:53:32.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:32.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-46 /etc/passwd 2024-09-10T19:53:33.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-47 /etc/passwd 2024-09-10T19:53:33.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-48 /etc/passwd 2024-09-10T19:53:33.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-49 /etc/passwd 2024-09-10T19:53:33.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-50 /etc/passwd 2024-09-10T19:53:33.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-51 /etc/passwd 2024-09-10T19:53:33.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-52 /etc/passwd 2024-09-10T19:53:33.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:33.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-53 /etc/passwd 2024-09-10T19:53:34.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-54 /etc/passwd 2024-09-10T19:53:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-55 /etc/passwd 2024-09-10T19:53:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.377 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-56 /etc/passwd 2024-09-10T19:53:34.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-57 /etc/passwd 2024-09-10T19:53:34.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-58 /etc/passwd 2024-09-10T19:53:34.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-59 /etc/passwd 2024-09-10T19:53:34.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:34.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-60 /etc/passwd 2024-09-10T19:53:35.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-61 /etc/passwd 2024-09-10T19:53:35.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-62 /etc/passwd 2024-09-10T19:53:35.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-63 /etc/passwd 2024-09-10T19:53:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-64 /etc/passwd 2024-09-10T19:53:35.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-65 /etc/passwd 2024-09-10T19:53:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:35.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-66 /etc/passwd 2024-09-10T19:53:36.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-67 /etc/passwd 2024-09-10T19:53:36.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-68 /etc/passwd 2024-09-10T19:53:36.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-69 /etc/passwd 2024-09-10T19:53:36.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-70 /etc/passwd 2024-09-10T19:53:36.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-71 /etc/passwd 2024-09-10T19:53:36.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-72 /etc/passwd 2024-09-10T19:53:36.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:36.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-73 /etc/passwd 2024-09-10T19:53:37.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-74 /etc/passwd 2024-09-10T19:53:37.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-75 /etc/passwd 2024-09-10T19:53:37.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-76 /etc/passwd 2024-09-10T19:53:37.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-77 /etc/passwd 2024-09-10T19:53:37.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-78 /etc/passwd 2024-09-10T19:53:37.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-79 /etc/passwd 2024-09-10T19:53:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-80 /etc/passwd 2024-09-10T19:53:38.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-81 /etc/passwd 2024-09-10T19:53:38.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-82 /etc/passwd 2024-09-10T19:53:38.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-83 /etc/passwd 2024-09-10T19:53:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-84 /etc/passwd 2024-09-10T19:53:38.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-85 /etc/passwd 2024-09-10T19:53:38.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:38.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-86 /etc/passwd 2024-09-10T19:53:39.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-87 /etc/passwd 2024-09-10T19:53:39.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-88 /etc/passwd 2024-09-10T19:53:39.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-89 /etc/passwd 2024-09-10T19:53:39.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-90 /etc/passwd 2024-09-10T19:53:39.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.568 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-91 /etc/passwd 2024-09-10T19:53:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-92 /etc/passwd 2024-09-10T19:53:39.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-93 /etc/passwd 2024-09-10T19:53:39.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:39.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-94 /etc/passwd 2024-09-10T19:53:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-95 /etc/passwd 2024-09-10T19:53:40.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-96 /etc/passwd 2024-09-10T19:53:40.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-97 /etc/passwd 2024-09-10T19:53:40.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-98 /etc/passwd 2024-09-10T19:53:40.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-99 /etc/passwd 2024-09-10T19:53:40.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:40.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-100 /etc/passwd 2024-09-10T19:53:41.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-101 /etc/passwd 2024-09-10T19:53:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-102 /etc/passwd 2024-09-10T19:53:41.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-103 /etc/passwd 2024-09-10T19:53:41.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-104 /etc/passwd 2024-09-10T19:53:41.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-105 /etc/passwd 2024-09-10T19:53:41.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-106 /etc/passwd 2024-09-10T19:53:41.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:41.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-107 /etc/passwd 2024-09-10T19:53:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-108 /etc/passwd 2024-09-10T19:53:42.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-109 /etc/passwd 2024-09-10T19:53:42.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-110 /etc/passwd 2024-09-10T19:53:42.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-111 /etc/passwd 2024-09-10T19:53:42.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-112 /etc/passwd 2024-09-10T19:53:42.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-113 /etc/passwd 2024-09-10T19:53:42.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:42.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-114 /etc/passwd 2024-09-10T19:53:43.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-115 /etc/passwd 2024-09-10T19:53:43.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-116 /etc/passwd 2024-09-10T19:53:43.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-117 /etc/passwd 2024-09-10T19:53:43.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-118 /etc/passwd 2024-09-10T19:53:43.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-119 /etc/passwd 2024-09-10T19:53:43.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-120 /etc/passwd 2024-09-10T19:53:43.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:43.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-121 /etc/passwd 2024-09-10T19:53:44.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-122 /etc/passwd 2024-09-10T19:53:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-123 /etc/passwd 2024-09-10T19:53:44.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-124 /etc/passwd 2024-09-10T19:53:44.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-125 /etc/passwd 2024-09-10T19:53:44.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-126 /etc/passwd 2024-09-10T19:53:44.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-127 /etc/passwd 2024-09-10T19:53:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:44.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-128 /etc/passwd 2024-09-10T19:53:45.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-129 /etc/passwd 2024-09-10T19:53:45.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-130 /etc/passwd 2024-09-10T19:53:45.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-131 /etc/passwd 2024-09-10T19:53:45.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-132 /etc/passwd 2024-09-10T19:53:45.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-133 /etc/passwd 2024-09-10T19:53:45.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:45.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-134 /etc/passwd 2024-09-10T19:53:46.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-135 /etc/passwd 2024-09-10T19:53:46.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-136 /etc/passwd 2024-09-10T19:53:46.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-137 /etc/passwd 2024-09-10T19:53:46.478 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-138 /etc/passwd 2024-09-10T19:53:46.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-139 /etc/passwd 2024-09-10T19:53:46.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-140 /etc/passwd 2024-09-10T19:53:46.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:46.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-141 /etc/passwd 2024-09-10T19:53:47.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-142 /etc/passwd 2024-09-10T19:53:47.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-143 /etc/passwd 2024-09-10T19:53:47.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.334 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-144 /etc/passwd 2024-09-10T19:53:47.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.485 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-145 /etc/passwd 2024-09-10T19:53:47.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-146 /etc/passwd 2024-09-10T19:53:47.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-147 /etc/passwd 2024-09-10T19:53:47.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:47.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-148 /etc/passwd 2024-09-10T19:53:48.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:48.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-149 /etc/passwd 2024-09-10T19:53:48.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:53:48.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-150 /etc/passwd 2024-09-10T19:53:48.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:53:48.400 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: jq '.pg_stats[0].up[]' 2024-09-10T19:53:48.765 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:53:48.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph osd out 1 0 2 2024-09-10T19:53:49.271 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2024-09-10T19:53:49.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:69: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:53:49.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: sleep 1 2024-09-10T19:53:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:76: _common_test: wait_for_clean 2024-09-10T19:53:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:53:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:53:50.286 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:53:50.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:53:50.287 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:53:50.287 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:53:50.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:53:50.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:53:50.287 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:53:50.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:53:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:53:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:53:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:53:50.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:53:50.382 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:53:50.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:50.902 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:53:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836530 2024-09-10T19:53:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836530 2024-09-10T19:53:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530' 2024-09-10T19:53:51.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:51.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:53:51.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673001 2024-09-10T19:53:51.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673001 2024-09-10T19:53:51.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673001' 2024-09-10T19:53:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:51.246 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:53:51.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509475 2024-09-10T19:53:51.359 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509475 2024-09-10T19:53:51.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509475' 2024-09-10T19:53:51.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:51.360 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:53:51.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345950 2024-09-10T19:53:51.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345950 2024-09-10T19:53:51.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509475 3-85899345950' 2024-09-10T19:53:51.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:51.514 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:53:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182417 2024-09-10T19:53:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182417 2024-09-10T19:53:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509475 3-85899345950 4-107374182417' 2024-09-10T19:53:51.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:53:51.696 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018892 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018892 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836530 1-42949673001 2-64424509475 3-85899345950 4-107374182417 5-128849018892' 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836530 2024-09-10T19:53:51.812 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:51.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:53:51.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836530 2024-09-10T19:53:51.813 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:51.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836530 2024-09-10T19:53:51.814 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836530 2024-09-10T19:53:51.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-10T19:53:51.815 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:53:52.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836530 2024-09-10T19:53:52.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:52.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673001 2024-09-10T19:53:52.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:52.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:53:52.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673001 2024-09-10T19:53:52.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:52.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673001 2024-09-10T19:53:52.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673001' 2024-09-10T19:53:52.439 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673001 2024-09-10T19:53:52.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:53:52.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673001 -lt 42949673001 2024-09-10T19:53:52.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:52.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509475 2024-09-10T19:53:52.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:52.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:53:52.968 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509475 2024-09-10T19:53:52.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:52.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509475 2024-09-10T19:53:52.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-10T19:53:52.970 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509475 2024-09-10T19:53:52.970 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:53:53.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-10T19:53:53.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:53.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345950 2024-09-10T19:53:53.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:53.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:53:53.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345950 2024-09-10T19:53:53.596 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:53.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345950 2024-09-10T19:53:53.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-09-10T19:53:53.597 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345950 2024-09-10T19:53:53.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:53:54.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345950 -lt 85899345950 2024-09-10T19:53:54.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:54.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182417 2024-09-10T19:53:54.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:54.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:53:54.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182417 2024-09-10T19:53:54.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:54.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182417 2024-09-10T19:53:54.242 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182417 2024-09-10T19:53:54.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-09-10T19:53:54.242 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:53:54.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182417 -lt 107374182417 2024-09-10T19:53:54.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:53:54.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018892 2024-09-10T19:53:54.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:53:54.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:53:54.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018892 2024-09-10T19:53:54.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:53:54.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018892 2024-09-10T19:53:54.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018892' 2024-09-10T19:53:54.795 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018892 2024-09-10T19:53:54.795 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:53:55.325 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018893 -lt 128849018892 2024-09-10T19:53:55.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:53:55.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:53:55.326 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:53:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T19:53:56.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:53:56.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:53:56.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:53:56.577 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:53:56.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:53:56.578 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:53:57.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:53:57.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:53:57.316 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:53:57.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:53:57.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:53:57.762 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:53:57.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:53:58.356 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:53:58.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=34343 2024-09-10T19:53:58.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 34343 '!=' null 2024-09-10T19:53:58.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:53:58.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:53:58.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:53:59.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:53:59.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:53:59.064 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:53:59.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:53:59.522 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:53:59.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:53:59.523 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:00.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:00.227 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:00.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=27932 2024-09-10T19:54:00.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 27932 '!=' null 2024-09-10T19:54:00.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:00.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:00.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:00.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:00.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:00.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:00.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:01.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:01.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:01.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:01.000 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:01.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:01.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:01.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:01.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:01.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:01.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:01.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:01.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=56021 2024-09-10T19:54:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 56021 '!=' null 2024-09-10T19:54:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:02.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:02.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:02.746 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:02.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:03.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:03.193 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:03.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:03.194 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:03.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:03.819 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:04.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=47909 2024-09-10T19:54:04.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 47909 '!=' null 2024-09-10T19:54:04.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:04.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:04.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:04.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:04.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:04.568 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:05.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:05.024 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:05.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:05.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:05.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:05.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:05.601 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:06.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=47909 2024-09-10T19:54:06.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 47909 '!=' null 2024-09-10T19:54:06.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:06.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:06.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:06.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:06.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:06.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:06.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:06.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:06.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:06.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:06.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:06.669 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:06.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:06.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:06.670 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:07.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:07.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:07.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:07.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:07.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:07.338 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:07.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=77721 2024-09-10T19:54:07.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 77721 '!=' null 2024-09-10T19:54:07.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:07.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:07.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:08.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:08.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:08.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:08.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:08.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:08.380 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:08.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:08.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:08.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:08.965 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:09.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=77721 2024-09-10T19:54:09.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 77721 '!=' null 2024-09-10T19:54:09.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:09.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:09.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:09.694 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:09.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:09.695 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:10.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:10.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:10.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:10.108 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:10.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:10.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:10.687 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=57492 2024-09-10T19:54:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 57492 '!=' null 2024-09-10T19:54:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:11.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:11.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:11.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:11.450 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:11.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:11.834 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:11.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:11.835 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:12.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:12.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:12.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=87302 2024-09-10T19:54:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 87302 '!=' null 2024-09-10T19:54:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:13.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:13.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:13.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:13.213 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:13.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:13.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:13.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:13.214 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:13.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:13.652 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:13.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:13.653 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:14.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:14.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:14.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:14.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:14.342 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:15.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=59621 2024-09-10T19:54:15.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 59621 '!=' null 2024-09-10T19:54:15.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:15.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:15.040 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:15.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:15.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:15.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:15.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:15.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:15.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:15.556 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:16.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:16.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:16.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:16.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:16.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:16.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:16.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:16.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:16.228 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:16.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=59621 2024-09-10T19:54:16.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 59621 '!=' null 2024-09-10T19:54:16.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:16.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:16.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:16.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:16.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:17.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:17.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:17.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:17.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:17.986 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:18.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=88367 2024-09-10T19:54:18.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 88367 '!=' null 2024-09-10T19:54:18.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:18.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:18.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:18.692 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:19.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:19.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:19.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:19.135 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:19.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:19.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:19.864 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:20.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=58556 2024-09-10T19:54:20.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 58556 '!=' null 2024-09-10T19:54:20.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:20.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:20.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:20.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:20.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:20.630 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:21.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:21.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:21.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:21.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:21.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:21.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:21.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:22.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=43118 2024-09-10T19:54:22.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 43118 '!=' null 2024-09-10T19:54:22.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:22.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:22.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:22.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:22.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:22.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:22.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:22.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:22.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:22.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:22.408 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:22.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:22.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:22.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:22.914 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:23.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:23.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:23.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:23.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:23.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:23.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:23.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:23.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:23.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=43119 2024-09-10T19:54:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 43119 '!=' null 2024-09-10T19:54:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:24.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:24.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:24.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:24.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:24.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:24.114 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:24.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:24.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:24.115 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:24.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:54:24.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:24.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:24.491 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:54:25.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:54:25.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:54:25.017 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:54:25.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=43119 2024-09-10T19:54:25.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 43119 '!=' null 2024-09-10T19:54:25.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:54:25.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:54:25.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:54:25.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:54:25.657 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:54:26.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:54:26.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:54:26.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:54:26.032 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:54:26.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T19:54:26.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:54:26.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:54:26.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:78: _common_test: flush_pg_stats 2024-09-10T19:54:26.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:54:26.557 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:54:27.011 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:54:27.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:54:27.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.012 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:54:27.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836540 2024-09-10T19:54:27.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836540 2024-09-10T19:54:27.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540' 2024-09-10T19:54:27.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.126 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:54:27.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673010 2024-09-10T19:54:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673010 2024-09-10T19:54:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673010' 2024-09-10T19:54:27.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.241 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:54:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509484 2024-09-10T19:54:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509484 2024-09-10T19:54:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673010 2-64424509484' 2024-09-10T19:54:27.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.357 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:54:27.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345959 2024-09-10T19:54:27.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345959 2024-09-10T19:54:27.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673010 2-64424509484 3-85899345959' 2024-09-10T19:54:27.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.472 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:54:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182426 2024-09-10T19:54:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182426 2024-09-10T19:54:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673010 2-64424509484 3-85899345959 4-107374182426' 2024-09-10T19:54:27.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:54:27.587 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018901 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018901 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836540 1-42949673010 2-64424509484 3-85899345959 4-107374182426 5-128849018901' 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836540 2024-09-10T19:54:27.702 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:27.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:54:27.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836540 2024-09-10T19:54:27.704 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:27.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836540 2024-09-10T19:54:27.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-09-10T19:54:27.705 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836540 2024-09-10T19:54:27.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:54:28.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836540 -lt 21474836540 2024-09-10T19:54:28.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:28.172 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673010 2024-09-10T19:54:28.173 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:28.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:54:28.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673010 2024-09-10T19:54:28.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:28.175 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673010 2024-09-10T19:54:28.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673010 2024-09-10T19:54:28.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673010' 2024-09-10T19:54:28.175 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:54:28.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673010 -lt 42949673010 2024-09-10T19:54:28.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:28.598 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509484 2024-09-10T19:54:28.599 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:28.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:54:28.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509484 2024-09-10T19:54:28.600 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:28.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509484 2024-09-10T19:54:28.601 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509484 2024-09-10T19:54:28.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-09-10T19:54:28.602 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:54:29.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509484 -lt 64424509484 2024-09-10T19:54:29.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:29.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345959 2024-09-10T19:54:29.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:29.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:54:29.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345959 2024-09-10T19:54:29.027 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:29.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345959 2024-09-10T19:54:29.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345959' 2024-09-10T19:54:29.028 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345959 2024-09-10T19:54:29.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:54:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345959 -lt 85899345959 2024-09-10T19:54:29.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:29.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182426 2024-09-10T19:54:29.451 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182426 2024-09-10T19:54:29.452 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:29.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182426 2024-09-10T19:54:29.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-09-10T19:54:29.453 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182426 2024-09-10T19:54:29.454 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:54:29.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182427 -lt 107374182426 2024-09-10T19:54:29.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:54:29.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018901 2024-09-10T19:54:29.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:54:29.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:54:29.842 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018901 2024-09-10T19:54:29.843 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:54:29.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018901 2024-09-10T19:54:29.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-09-10T19:54:29.844 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018901 2024-09-10T19:54:29.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:54:30.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018902 -lt 128849018901 2024-09-10T19:54:30.295 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:54:30.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: jq '.pg_stats[0].up_primary' 2024-09-10T19:54:30.661 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: newprimary=4 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:81: _common_test: kill_daemons 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:54:30.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: ERRORS=0 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: tee td/osd-backfill-recovery-log/result.log 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T19:54:35.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=4 2024-09-10T19:54:35.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T19:54:35.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:54:35.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "34'150", 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "34'148", 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e9c179f9:::obj-149:head", 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'149", 2024-09-10T19:54:37.597 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4821.0:1", 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:53:48.150465+0000", 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:54:37.598 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cb39fd60:::obj-150:head", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'150", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4824.0:1", 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:54:37.599 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:53:48.284466+0000", 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:54:37.600 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [ 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4377.0:1", 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'1", 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "1", 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.601 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4380.0:1", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'2", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "2", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4383.0:1", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'3", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "3", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4386.0:1", 2024-09-10T19:54:37.602 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'4", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "4", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4389.0:1", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'5", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "5", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4392.0:1", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'6", 2024-09-10T19:54:37.603 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "6", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4395.0:1", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'7", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "7", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4398.0:1", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'8", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "8", 2024-09-10T19:54:37.604 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4401.0:1", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'9", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "9", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4404.0:1", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'10", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "10", 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.605 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4407.0:1", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'11", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "11", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4410.0:1", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'12", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "12", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4413.0:1", 2024-09-10T19:54:37.606 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'13", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "13", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4416.0:1", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'14", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "14", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4419.0:1", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'15", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "15", 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.607 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4422.0:1", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'16", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "16", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4425.0:1", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'17", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "17", 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.608 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4428.0:1", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'18", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "18", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4431.0:1", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'19", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "19", 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.609 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4434.0:1", 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'20", 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "20", 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4437.0:1", 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'21", 2024-09-10T19:54:37.610 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "21", 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4440.0:1", 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'22", 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "22", 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4443.0:1", 2024-09-10T19:54:37.611 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'23", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "23", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4446.0:1", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'24", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "24", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4449.0:1", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'25", 2024-09-10T19:54:37.612 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "25", 2024-09-10T19:54:37.613 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.613 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.613 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4452.0:1", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'26", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "26", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4455.0:1", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'27", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "27", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4458.0:1", 2024-09-10T19:54:37.614 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'28", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "28", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4461.0:1", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'29", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "29", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4464.0:1", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'30", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "30", 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.615 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4467.0:1", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'31", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "31", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4470.0:1", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'32", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "32", 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.616 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4473.0:1", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'33", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "33", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4476.0:1", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'34", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "34", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4479.0:1", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'35", 2024-09-10T19:54:37.617 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "35", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4482.0:1", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'36", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "36", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4485.0:1", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'37", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "37", 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.618 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4488.0:1", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'38", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "38", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4491.0:1", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'39", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "39", 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.619 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4494.0:1", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'40", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "40", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4497.0:1", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'41", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "41", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4500.0:1", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'42", 2024-09-10T19:54:37.620 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "42", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4503.0:1", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'43", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "43", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4506.0:1", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'44", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "44", 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.621 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4509.0:1", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'45", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "45", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4512.0:1", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'46", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "46", 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.622 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4515.0:1", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'47", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "47", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4518.0:1", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'48", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "48", 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.623 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4521.0:1", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'49", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "49", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4524.0:1", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'50", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "50", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4527.0:1", 2024-09-10T19:54:37.624 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'51", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "51", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4530.0:1", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'52", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "52", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4533.0:1", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'53", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "53", 2024-09-10T19:54:37.625 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4536.0:1", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'54", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "54", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4539.0:1", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'55", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "55", 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.626 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4542.0:1", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'56", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "56", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4545.0:1", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'57", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "57", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4548.0:1", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'58", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "58", 2024-09-10T19:54:37.627 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4551.0:1", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'59", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "59", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4554.0:1", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'60", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "60", 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.628 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4557.0:1", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'61", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "61", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4560.0:1", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'62", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "62", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4563.0:1", 2024-09-10T19:54:37.629 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'63", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "63", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4566.0:1", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'64", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "64", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4569.0:1", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'65", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "65", 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.630 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4572.0:1", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'66", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "66", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4575.0:1", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'67", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "67", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4578.0:1", 2024-09-10T19:54:37.631 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'68", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "68", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4581.0:1", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'69", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "69", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4584.0:1", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'70", 2024-09-10T19:54:37.632 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "70", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4587.0:1", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'71", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "71", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4590.0:1", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'72", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "72", 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.633 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4593.0:1", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'73", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "73", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4596.0:1", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'74", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "74", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4599.0:1", 2024-09-10T19:54:37.634 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'75", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "75", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4602.0:1", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'76", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "76", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4605.0:1", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'77", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "77", 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.635 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4608.0:1", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'78", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "78", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4611.0:1", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'79", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "79", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4614.0:1", 2024-09-10T19:54:37.636 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'80", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "80", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4617.0:1", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'81", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "81", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4620.0:1", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'82", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "82", 2024-09-10T19:54:37.637 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4623.0:1", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'83", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "83", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4626.0:1", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'84", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "84", 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.638 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4629.0:1", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'85", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "85", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4632.0:1", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'86", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "86", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4635.0:1", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'87", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "87", 2024-09-10T19:54:37.639 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4638.0:1", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'88", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "88", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4641.0:1", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'89", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "89", 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.640 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4644.0:1", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'90", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "90", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4647.0:1", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'91", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "91", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4650.0:1", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'92", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "92", 2024-09-10T19:54:37.641 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4653.0:1", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'93", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "93", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4656.0:1", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'94", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "94", 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.642 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4659.0:1", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'95", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "95", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4662.0:1", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'96", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "96", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4665.0:1", 2024-09-10T19:54:37.643 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'97", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "97", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4668.0:1", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'98", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "98", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4671.0:1", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'99", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "99", 2024-09-10T19:54:37.644 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4674.0:1", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'100", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "100", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4677.0:1", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'101", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "101", 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.645 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4680.0:1", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'102", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "102", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4683.0:1", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'103", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "103", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4686.0:1", 2024-09-10T19:54:37.646 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'104", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "104", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4689.0:1", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'105", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "105", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4692.0:1", 2024-09-10T19:54:37.647 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'106", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "106", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4695.0:1", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'107", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "107", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4698.0:1", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'108", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "108", 2024-09-10T19:54:37.648 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4701.0:1", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'109", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "109", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4704.0:1", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'110", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "110", 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.649 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4707.0:1", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'111", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "111", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4710.0:1", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'112", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "112", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4713.0:1", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'113", 2024-09-10T19:54:37.650 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "113", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4716.0:1", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'114", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "114", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4719.0:1", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'115", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "115", 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.651 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4722.0:1", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'116", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "116", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4725.0:1", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'117", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "117", 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.652 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4728.0:1", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'118", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "118", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4731.0:1", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'119", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "119", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4734.0:1", 2024-09-10T19:54:37.653 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'120", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "120", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4737.0:1", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'121", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "121", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4740.0:1", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'122", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "122", 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.654 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4743.0:1", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'123", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "123", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4746.0:1", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'124", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "124", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4749.0:1", 2024-09-10T19:54:37.655 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'125", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "125", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4752.0:1", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'126", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "126", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4755.0:1", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'127", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "127", 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.656 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4758.0:1", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'128", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "128", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4761.0:1", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'129", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "129", 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.657 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4764.0:1", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'130", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "130", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4767.0:1", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'131", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "131", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4770.0:1", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'132", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "132", 2024-09-10T19:54:37.658 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4773.0:1", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'133", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "133", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4776.0:1", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'134", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "134", 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.659 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4779.0:1", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'135", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "135", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4782.0:1", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'136", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "136", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4785.0:1", 2024-09-10T19:54:37.660 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'137", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "137", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4788.0:1", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'138", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "138", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4791.0:1", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'139", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "139", 2024-09-10T19:54:37.661 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4794.0:1", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'140", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "140", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4797.0:1", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'141", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "141", 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.662 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4800.0:1", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'142", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "142", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4803.0:1", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'143", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "143", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4806.0:1", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'144", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "144", 2024-09-10T19:54:37.663 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4809.0:1", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'145", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "145", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4812.0:1", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'146", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "146", 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.664 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.665 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4815.0:1", 2024-09-10T19:54:37.665 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'147", 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "147", 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4818.0:1", 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'148", 2024-09-10T19:54:37.857 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "148", 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:54:37.858 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:54:37.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:54:37.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: LOGLEN=2 2024-09-10T19:54:37.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:86: _common_test: '[' 2 '!=' 2 ']' 2024-09-10T19:54:37.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:54:37.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: DUPSLEN=148 2024-09-10T19:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:91: _common_test: '[' 148 '!=' 148 ']' 2024-09-10T19:54:37.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=100 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'150 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'149 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'1 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'2 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'3 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'4 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'5 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'6 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'7 2024-09-10T19:54:37.872 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'8 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'9 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'10 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'11 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'12 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'13 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'14 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'15 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'16 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'17 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'18 2024-09-10T19:54:37.873 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'19 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'20 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'21 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'22 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'23 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'24 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'25 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'26 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'27 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'28 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'29 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'30 2024-09-10T19:54:37.874 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'31 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'32 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'33 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'34 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'35 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'36 2024-09-10T19:54:37.875 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'37 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'38 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'39 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'40 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'41 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'42 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'43 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'44 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'45 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'46 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'47 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'48 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'49 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'50 2024-09-10T19:54:37.876 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'51 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'52 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'53 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'54 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'55 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'56 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'57 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'58 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'59 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'60 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'61 2024-09-10T19:54:37.877 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'62 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'63 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'64 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'65 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'66 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'67 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'68 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'69 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'70 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'71 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'72 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'73 2024-09-10T19:54:37.878 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'74 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'75 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'76 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'77 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'78 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'79 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'80 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'81 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'82 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'83 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'84 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'85 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'86 2024-09-10T19:54:37.879 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'87 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'88 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'89 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'90 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'91 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'92 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'93 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'94 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'95 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'96 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'97 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'98 2024-09-10T19:54:37.880 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'99 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'100 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'101 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'102 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'103 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'104 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'105 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'106 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'107 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'108 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'109 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'110 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'111 2024-09-10T19:54:37.881 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'112 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'113 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'114 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'115 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'116 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'117 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'118 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'119 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'120 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'121 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'122 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'123 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'124 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'125 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'126 2024-09-10T19:54:37.882 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'127 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'128 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'129 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'130 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'131 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'132 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'133 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'134 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'135 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'136 2024-09-10T19:54:37.883 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'137 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'138 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'139 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'140 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to END max 2 dups.size()=148 other.dups.size()=100 2024-09-10T19:54:37.884 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=100 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'150 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'149 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'1 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'2 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'3 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'4 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'5 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'6 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'7 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'8 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'9 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'10 2024-09-10T19:54:37.885 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'11 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'12 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'13 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'14 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'15 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'16 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'17 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'18 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'19 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'20 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'21 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'22 2024-09-10T19:54:37.886 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'23 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'24 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'25 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'26 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'27 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'28 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'29 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'30 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'31 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'32 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'33 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'34 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'35 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'36 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'37 2024-09-10T19:54:37.887 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'38 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'39 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'40 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'41 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'42 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'43 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'44 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'45 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'46 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'47 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'48 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'49 2024-09-10T19:54:37.888 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'50 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'51 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'52 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'53 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'54 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'55 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'56 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'57 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'58 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'59 2024-09-10T19:54:37.889 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'60 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'61 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'62 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'63 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'64 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'65 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'66 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'67 2024-09-10T19:54:37.890 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'68 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'69 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'70 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'71 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'72 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'73 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'74 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'75 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'76 2024-09-10T19:54:37.891 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'77 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'78 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'79 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'80 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'81 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'82 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'83 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'84 2024-09-10T19:54:37.892 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'85 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'86 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'87 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'88 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'89 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'90 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'91 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'92 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'93 2024-09-10T19:54:37.893 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'94 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'95 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'96 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'97 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'98 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'99 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'100 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'101 2024-09-10T19:54:37.894 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'102 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'103 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'104 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'105 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'106 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'107 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'108 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'109 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'110 2024-09-10T19:54:37.895 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'111 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'112 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'113 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'114 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'115 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'116 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'117 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'118 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'119 2024-09-10T19:54:37.896 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'120 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'121 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'122 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'123 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'124 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'125 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'126 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'127 2024-09-10T19:54:37.897 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'128 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'129 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'130 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'131 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'132 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'133 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'134 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'135 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'136 2024-09-10T19:54:37.898 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'137 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'138 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'139 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'140 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:54:37.899 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to END max 2 dups.size()=148 other.dups.size()=100 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to max 2 dups.size()=0 other.dups.size()=100 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'150 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to copy log version 34'149 2024-09-10T19:54:37.900 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'1 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'2 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'3 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'4 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'5 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'6 2024-09-10T19:54:37.901 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'7 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'8 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'9 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'10 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'11 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'12 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'13 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'14 2024-09-10T19:54:37.902 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'15 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'16 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'17 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'18 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'19 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'20 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'21 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'22 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'23 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'24 2024-09-10T19:54:37.903 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'25 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'26 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'27 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'28 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'29 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'30 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'31 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'32 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'33 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'34 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'35 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'36 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'37 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'38 2024-09-10T19:54:37.904 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'39 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'40 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'41 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'42 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'43 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'44 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'45 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'46 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'47 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'48 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'49 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'50 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'51 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'52 2024-09-10T19:54:37.905 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'53 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'54 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'55 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'56 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'57 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'58 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'59 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'60 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'61 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'62 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'63 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'64 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'65 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'66 2024-09-10T19:54:37.906 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'67 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'68 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'69 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'70 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'71 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'72 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'73 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'74 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'75 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'76 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'77 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'78 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'79 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'80 2024-09-10T19:54:37.907 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'81 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'82 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'83 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'84 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'85 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'86 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'87 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'88 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'89 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'90 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'91 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'92 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'93 2024-09-10T19:54:37.908 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'94 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'95 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'96 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'97 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'98 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'99 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup version 34'100 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'101 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'102 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'103 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'104 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'105 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'106 2024-09-10T19:54:37.909 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'107 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'108 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'109 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'110 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'111 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'112 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'113 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'114 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'115 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'116 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'117 2024-09-10T19:54:37.910 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'118 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'119 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'120 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'121 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'122 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'123 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'124 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'125 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'126 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'127 2024-09-10T19:54:37.911 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'128 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'129 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'130 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'131 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'132 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'133 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'134 2024-09-10T19:54:37.912 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'135 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'136 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'137 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'138 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'139 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'140 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:54:37.913 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:53:51.331+0000 7f2a93dbd640 20 copy_up_to END max 2 dups.size()=148 other.dups.size()=100 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_after v 34'150 dups.size()=0 other.dups.size()=100 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'1 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'2 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'3 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'4 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'5 2024-09-10T19:54:37.914 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'6 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'7 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'8 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'9 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'10 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'11 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'12 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'13 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'14 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'15 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'16 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'17 2024-09-10T19:54:37.915 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'18 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'19 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'20 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'21 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'22 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'23 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'24 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'25 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'26 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'27 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'28 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'29 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'30 2024-09-10T19:54:37.916 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'31 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'32 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'33 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'34 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'35 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'36 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'37 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'38 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'39 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'40 2024-09-10T19:54:37.917 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'41 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'42 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'43 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'44 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'45 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'46 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'47 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'48 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'49 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'50 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'51 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'52 2024-09-10T19:54:37.918 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'53 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'54 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'55 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'56 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'57 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'58 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'59 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'60 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'61 2024-09-10T19:54:37.920 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'62 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'63 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'64 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'65 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'66 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'67 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'68 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'69 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'70 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'71 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'72 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'73 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'74 2024-09-10T19:54:37.921 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'75 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'76 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'77 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'78 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'79 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'80 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'81 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'82 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'83 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'84 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'85 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'86 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'87 2024-09-10T19:54:37.922 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'88 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'89 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'90 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'91 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'92 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'93 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'94 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'95 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'96 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'97 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'98 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'99 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup version 34'100 2024-09-10T19:54:37.923 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'101 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'102 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'103 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'104 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'105 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'106 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'107 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'108 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'109 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'110 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'111 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'112 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'113 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'114 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'115 2024-09-10T19:54:37.924 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'116 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'117 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'118 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'119 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'120 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'121 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'122 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'123 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'124 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'125 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'126 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'127 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'128 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'129 2024-09-10T19:54:37.925 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'130 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'131 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'132 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'133 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'134 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'135 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'136 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'137 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'138 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'139 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'140 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'142 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'143 2024-09-10T19:54:37.926 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'144 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'145 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'146 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'147 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'148 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'149 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_up_to/copy_after copy dup from log version 34'150 2024-09-10T19:54:37.927 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:54:21.149+0000 7f2a955c0640 20 copy_after END v 34'150 dups.size()=150 other.dups.size()=100 2024-09-10T19:54:37.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2024-09-10T19:54:37.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:97: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:54:37.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:35: run: teardown td/osd-backfill-recovery-log 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:54:37.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:54:37.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:54:37.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:54:37.956 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:37.957 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:37.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:54:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:54:37.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: for func in $funcs 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: setup td/osd-backfill-recovery-log 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-recovery-log 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:54:37.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:54:37.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:54:37.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:54:37.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:54:37.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:54:37.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:54:37.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:54:37.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:54:37.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:54:37.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:54:37.964 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:54:37.964 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:54:37.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:54:37.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:54:37.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:54:37.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:54:37.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:37.967 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:37.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:54:37.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:54:37.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:54:37.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-recovery-log 2024-09-10T19:54:37.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:54:37.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:37.969 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:37.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53443 2024-09-10T19:54:37.972 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:54:37.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T19:54:37.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:54:37.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2024-09-10T19:54:37.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: TEST_recovery_1 td/osd-backfill-recovery-log 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:122: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:124: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local loglen=40 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local dupslen=0 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local objects=40 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:46: _common_test: local moreobjects=0 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:48: _common_test: local OSDS=6 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mon td/osd-backfill-recovery-log a 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:54:37.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:54:37.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:54:37.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-recovery-log/a 2024-09-10T19:54:37.974 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2024-09-10T19:54:38.444 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:54:38.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:54:38.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:54:38.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:54:38.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:54:38.638 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:38.639 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:54:38.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:54:38.640 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get fsid 2024-09-10T19:54:38.720 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:54:38.721 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get mon_host 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: run_mgr td/osd-backfill-recovery-log x 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-recovery-log/x 2024-09-10T19:54:38.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:39.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:54:39.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:54:39.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:54:39.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export CEPH_ARGS 2024-09-10T19:54:39.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:54:39.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:54:39.682 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: expr 6 - 1 2024-09-10T19:54:39.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: seq 0 5 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 0 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:54:39.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:54:39.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2024-09-10T19:54:39.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:54:39.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b723782e-31e9-48eb-8992-ec1e1b6b9857 2024-09-10T19:54:39.690 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 b723782e-31e9-48eb-8992-ec1e1b6b9857 2024-09-10T19:54:39.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b723782e-31e9-48eb-8992-ec1e1b6b9857' 2024-09-10T19:54:39.691 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:54:39.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/o+BmcVoSKhAAakVlD1KRLuYtt+1fLOWcmg== 2024-09-10T19:54:39.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/o+BmcVoSKhAAakVlD1KRLuYtt+1fLOWcmg=="}' 2024-09-10T19:54:39.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b723782e-31e9-48eb-8992-ec1e1b6b9857 -i td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:54:39.978 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:54:39.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:54:39.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/o+BmcVoSKhAAakVlD1KRLuYtt+1fLOWcmg== --osd-uuid b723782e-31e9-48eb-8992-ec1e1b6b9857 2024-09-10T19:54:40.024 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:40.021+0000 7fe268fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:40.047 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:40.044+0000 7fe268fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:40.048 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:40.046+0000 7fe268fb2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:40.048 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:40.046+0000 7fe268fb2640 -1 bdev(0x55f0214e9c00 td/osd-backfill-recovery-log/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:54:40.048 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:40.046+0000 7fe268fb2640 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2024-09-10T19:54:45.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2024-09-10T19:54:45.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:54:45.529 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:54:45.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:54:45.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:54:46.079 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:54:46.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:54:46.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:54:46.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:54:46.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:54:46.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:54:46.111 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:46.109+0000 7f1678260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:46.113 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:46.111+0000 7f1678260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:46.114 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:46.112+0000 7f1678260640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:54:46.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:46.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:54:46.499 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:54:46.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:46.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:46.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:47.464 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:47.462+0000 7f1678260640 -1 Falling back to public interface 2024-09-10T19:54:47.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:47.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:47.917 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:54:47.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:54:47.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:47.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:48.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:49.071 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:54:49.068+0000 7f1678260640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:54:49.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:49.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:49.360 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:54:49.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:54:49.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:49.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:49.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:50.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:50.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:50.863 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:54:50.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:54:50.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:50.864 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:51.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:52.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:52.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:52.248 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:54:52.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:54:52.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:52.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:52.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:53.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:53.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:53.646 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:54:53.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:54:53.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:53.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:54.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:55.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:55.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:55.165 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:54:55.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:54:55.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:55.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:55.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:56.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:56.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:56.564 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:54:56.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:54:56.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:56.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:57.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:58.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:58.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:58.019 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:54:58.019 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:54:58.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:58.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:58.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:54:59.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:54:59.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:54:59.443 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:54:59.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:54:59.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:54:59.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:54:59.937 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:00.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:00.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:00.939 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:55:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:55:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:00.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:01.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:02.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:02.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:02.594 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:55:02.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:55:02.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:02.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:03.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:04.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:04.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:04.179 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:55:04.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:55:04.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:04.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:05.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:06.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:06.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:06.214 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:55:06.214 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:55:06.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:06.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:06.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:07.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:07.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:07.970 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:55:07.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:55:07.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:07.971 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:09.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:10.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:10.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:10.071 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:55:10.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:55:10.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:10.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:11.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:12.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:12.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:12.226 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:55:12.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:55:12.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:12.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:55:12.639 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1858135360,v1:127.0.0.1:6803/1858135360] [v2:127.0.0.1:6804/1858135360,v1:127.0.0.1:6805/1858135360] exists,up b723782e-31e9-48eb-8992-ec1e1b6b9857 2024-09-10T19:55:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:55:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:55:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:55:12.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 1 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:55:12.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:55:12.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2024-09-10T19:55:12.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:55:12.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=80580c50-c4b6-41af-be7e-3087000f5770 2024-09-10T19:55:12.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 80580c50-c4b6-41af-be7e-3087000f5770' 2024-09-10T19:55:12.647 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 80580c50-c4b6-41af-be7e-3087000f5770 2024-09-10T19:55:12.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:55:12.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAgpOBmu/91JxAA4TrpN/T4Vc+fpNaSYbECEQ== 2024-09-10T19:55:12.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAgpOBmu/91JxAA4TrpN/T4Vc+fpNaSYbECEQ=="}' 2024-09-10T19:55:12.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 80580c50-c4b6-41af-be7e-3087000f5770 -i td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:55:13.197 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:55:13.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:55:13.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgpOBmu/91JxAA4TrpN/T4Vc+fpNaSYbECEQ== --osd-uuid 80580c50-c4b6-41af-be7e-3087000f5770 2024-09-10T19:55:13.243 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:13.240+0000 7f18b222c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:13.245 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:13.242+0000 7f18b222c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:13.246 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:13.243+0000 7f18b222c640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:13.246 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:13.243+0000 7f18b222c640 -1 bdev(0x55a890751c00 td/osd-backfill-recovery-log/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:55:13.247 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:13.243+0000 7f18b222c640 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2024-09-10T19:55:19.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2024-09-10T19:55:19.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:55:19.061 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:55:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:55:19.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:55:19.613 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:55:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:55:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:55:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:55:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:55:19.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:55:19.647 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:19.644+0000 7faad4e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:19.650 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:19.647+0000 7faad4e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:19.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:19.648+0000 7faad4e15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:20.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:55:20.083 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:55:20.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:20.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:20.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:20.764 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:20.761+0000 7faad4e15640 -1 Falling back to public interface 2024-09-10T19:55:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:21.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:21.509 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:55:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:55:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:21.509 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:21.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:22.601+0000 7faad4e15640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:55:22.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:22.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:22.912 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:55:22.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:55:22.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:22.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:23.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:24.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:24.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:24.416 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:55:24.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:55:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:24.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:24.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:25.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:25.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:25.837 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:55:25.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:55:25.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:25.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:26.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:27.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:27.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:27.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:55:27.301 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:55:27.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:27.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:27.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:28.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:28.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:28.774 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:55:28.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:55:28.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:28.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:29.193 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:30.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:30.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:30.195 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:55:30.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:55:30.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:30.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:30.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:31.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:31.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:31.629 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:55:31.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:55:31.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:31.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:32.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:33.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:33.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:33.112 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:55:33.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:55:33.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:33.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:33.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:34.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:34.545 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:55:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:55:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:34.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:35.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:36.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:36.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:36.118 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:55:36.118 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:55:36.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:36.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:37.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:38.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:38.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:38.155 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:55:38.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:55:38.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:38.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:38.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:39.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:39.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:39.933 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:55:39.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:55:39.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:39.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:40.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:41.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:41.809 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:55:41.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:55:41.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:41.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:42.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:43.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:43.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:43.654 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:55:43.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:55:43.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:43.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:44.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:45.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:45.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:45.448 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:55:45.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:55:45.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:45.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:45.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:46.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:46.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:46.984 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:55:46.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:55:46.984 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:46.985 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:55:47.412 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/223261326,v1:127.0.0.1:6811/223261326] [v2:127.0.0.1:6812/223261326,v1:127.0.0.1:6813/223261326] exists,up 80580c50-c4b6-41af-be7e-3087000f5770 2024-09-10T19:55:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:55:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:55:47.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 2 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:55:47.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:55:47.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:55:47.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e778b7cb-6bd1-456c-80d2-bc057e954921 2024-09-10T19:55:47.420 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 e778b7cb-6bd1-456c-80d2-bc057e954921 2024-09-10T19:55:47.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 e778b7cb-6bd1-456c-80d2-bc057e954921' 2024-09-10T19:55:47.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:55:47.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBDpOBmxPXZGRAAYQ+UOYJ3HCMYOb3vJyvFgA== 2024-09-10T19:55:47.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBDpOBmxPXZGRAAYQ+UOYJ3HCMYOb3vJyvFgA=="}' 2024-09-10T19:55:47.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e778b7cb-6bd1-456c-80d2-bc057e954921 -i td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:55:47.935 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:55:47.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:55:47.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDpOBmxPXZGRAAYQ+UOYJ3HCMYOb3vJyvFgA== --osd-uuid e778b7cb-6bd1-456c-80d2-bc057e954921 2024-09-10T19:55:47.983 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:47.980+0000 7faf8661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:47.982+0000 7faf8661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:47.985 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:47.983+0000 7faf8661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:47.984+0000 7faf8661d640 -1 bdev(0x555689c7bc00 td/osd-backfill-recovery-log/2/block) open stat got: (1) Operation not permitted 2024-09-10T19:55:47.986 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:47.984+0000 7faf8661d640 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2024-09-10T19:55:53.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2024-09-10T19:55:53.920 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:55:53.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T19:55:53.921 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T19:55:53.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:55:54.455 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T19:55:54.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T19:55:54.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:55:54.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:55:54.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:55:54.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:55:54.489 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:54.486+0000 7f41a2f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:54.495 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:54.493+0000 7f41a2f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:54.496 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:54.494+0000 7f41a2f45640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:55:54.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T19:55:54.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:55:54.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T19:55:54.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:55:54.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:55:54.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:54.894 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:55:54.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:55:54.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:54.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:55:55.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:55.581 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:55.579+0000 7f41a2f45640 -1 Falling back to public interface 2024-09-10T19:55:56.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:56.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:56.320 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:55:56.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:55:56.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:56.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:55:56.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:57.666 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:55:57.664+0000 7f41a2f45640 -1 osd.2 0 log_to_monitors true 2024-09-10T19:55:57.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:57.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:57.755 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:55:57.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:55:57.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:57.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:55:58.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:55:59.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:55:59.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:55:59.303 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:55:59.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:55:59.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:55:59.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:55:59.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:00.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:00.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:00.789 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:56:00.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:56:00.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:00.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:01.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:02.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:02.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:02.204 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:56:02.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:56:02.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:02.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:02.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:03.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:03.644 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:56:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:56:03.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:03.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:04.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:05.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:05.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:05.060 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:56:05.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:56:05.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:05.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:05.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:06.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:06.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:06.608 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:56:06.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:56:06.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:06.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:07.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:08.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:08.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:08.006 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:56:08.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:56:08.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:08.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:08.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:09.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:09.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:09.445 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:56:09.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:56:09.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:09.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:10.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:11.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:11.170 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:11.170 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:56:11.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:56:11.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:11.171 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:11.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:12.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:12.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:12.875 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:56:12.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:56:12.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:12.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:13.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:14.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:14.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:14.644 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:56:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:56:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:15.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:16.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:16.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:16.740 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:56:16.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:56:16.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:16.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:17.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:18.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:18.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:18.527 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:56:18.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:56:18.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:18.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:19.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:20.226 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:20.226 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:56:20.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:56:20.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:20.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:56:20.655 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2081521699,v1:127.0.0.1:6819/2081521699] [v2:127.0.0.1:6820/2081521699,v1:127.0.0.1:6821/2081521699] exists,up e778b7cb-6bd1-456c-80d2-bc057e954921 2024-09-10T19:56:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:56:20.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 3 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:56:20.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:56:20.661 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:56:20.662 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2024-09-10T19:56:20.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:56:20.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=36a86fa1-09db-49cf-b7a8-cdf3b783054b 2024-09-10T19:56:20.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 36a86fa1-09db-49cf-b7a8-cdf3b783054b' 2024-09-10T19:56:20.663 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 36a86fa1-09db-49cf-b7a8-cdf3b783054b 2024-09-10T19:56:20.663 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:56:20.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkpOBmb8FgKBAA0J7JDl9AM6yRj7w2Hasysg== 2024-09-10T19:56:20.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkpOBmb8FgKBAA0J7JDl9AM6yRj7w2Hasysg=="}' 2024-09-10T19:56:20.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 36a86fa1-09db-49cf-b7a8-cdf3b783054b -i td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:56:21.150 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:56:21.161 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2024-09-10T19:56:21.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkpOBmb8FgKBAA0J7JDl9AM6yRj7w2Hasysg== --osd-uuid 36a86fa1-09db-49cf-b7a8-cdf3b783054b 2024-09-10T19:56:21.197 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:21.195+0000 7f7c76ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:21.224 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:21.222+0000 7f7c76ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:21.225 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:21.223+0000 7f7c76ab6640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:21.226 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:21.223+0000 7f7c76ab6640 -1 bdev(0x5558b2669c00 td/osd-backfill-recovery-log/3/block) open stat got: (1) Operation not permitted 2024-09-10T19:56:21.226 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:21.223+0000 7f7c76ab6640 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2024-09-10T19:56:27.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2024-09-10T19:56:27.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:56:27.044 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-10T19:56:27.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T19:56:27.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:56:27.589 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-10T19:56:27.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T19:56:27.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:56:27.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:56:27.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:56:27.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:56:27.622 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:27.620+0000 7f01a661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:27.643 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:27.641+0000 7f01a661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:27.644 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:27.642+0000 7f01a661d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:56:28.075 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:56:28.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:28.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:28.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:28.744 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:28.742+0000 7f01a661d640 -1 Falling back to public interface 2024-09-10T19:56:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:29.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:29.494 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:56:29.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:56:29.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:29.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:29.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:30.588 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:30.586+0000 7f01a661d640 -1 osd.3 0 log_to_monitors true 2024-09-10T19:56:30.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:30.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:30.904 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:56:30.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:56:30.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:30.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:31.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:32.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:32.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:32.400 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:56:32.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:56:32.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:32.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:32.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:33.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:33.814 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:56:33.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:56:33.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:33.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:34.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:35.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:35.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:35.247 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:56:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:56:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:35.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:35.696 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:36.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:36.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:36.697 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:56:36.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:56:36.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:36.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:37.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:38.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:38.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:38.107 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:56:38.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:56:38.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:38.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:38.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:39.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:39.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:39.599 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:56:39.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:56:39.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:39.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:40.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:41.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:41.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:41.082 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:56:41.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:56:41.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:41.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:41.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:42.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:42.494 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:56:42.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:56:42.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:42.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:43.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:44.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:44.032 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:56:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:56:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:44.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:44.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:45.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:45.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:45.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:56:45.785 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:56:45.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:45.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:46.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:47.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:47.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:47.738 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:56:47.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:56:47.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:47.738 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:48.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:49.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:49.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:49.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:56:49.185 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:56:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:49.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:49.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:50.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:50.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:50.957 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:56:50.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:56:50.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:50.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:51.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:52.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:52.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:52.776 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:56:52.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:56:52.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:52.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:53.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:56:54.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:56:54.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:56:54.566 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:56:54.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:56:54.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:56:54.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T19:56:54.955 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1265470239,v1:127.0.0.1:6827/1265470239] [v2:127.0.0.1:6828/1265470239,v1:127.0.0.1:6829/1265470239] exists,up 36a86fa1-09db-49cf-b7a8-cdf3b783054b 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 4 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:56:54.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:56:54.960 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:56:54.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd4c87c8-cbaa-41c2-b8f9-c568f6055f21 2024-09-10T19:56:54.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 fd4c87c8-cbaa-41c2-b8f9-c568f6055f21' 2024-09-10T19:56:54.963 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 fd4c87c8-cbaa-41c2-b8f9-c568f6055f21 2024-09-10T19:56:54.963 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:56:54.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCGpOBmWoY5OhAAbgzmqzKxe2hYkLdiSyDnKA== 2024-09-10T19:56:54.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCGpOBmWoY5OhAAbgzmqzKxe2hYkLdiSyDnKA=="}' 2024-09-10T19:56:54.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd4c87c8-cbaa-41c2-b8f9-c568f6055f21 -i td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:56:55.478 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:56:55.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2024-09-10T19:56:55.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGpOBmWoY5OhAAbgzmqzKxe2hYkLdiSyDnKA== --osd-uuid fd4c87c8-cbaa-41c2-b8f9-c568f6055f21 2024-09-10T19:56:55.525 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:55.523+0000 7f0ed3e39640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:55.527 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:55.525+0000 7f0ed3e39640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:55.528 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:55.526+0000 7f0ed3e39640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:56:55.528 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:55.526+0000 7f0ed3e39640 -1 bdev(0x55b081c1bc00 td/osd-backfill-recovery-log/4/block) open stat got: (1) Operation not permitted 2024-09-10T19:56:55.528 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:56:55.526+0000 7f0ed3e39640 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2024-09-10T19:57:02.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2024-09-10T19:57:02.656 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:57:02.657 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-10T19:57:02.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T19:57:02.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:57:03.191 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-10T19:57:03.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T19:57:03.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:57:03.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:57:03.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:57:03.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:57:03.224 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:03.221+0000 7f09c7ad4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:03.239 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:03.236+0000 7f09c7ad4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:03.240 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:03.237+0000 7f09c7ad4640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:57:03.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:03.616 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:57:03.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:57:03.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:03.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:04.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:04.584 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:04.581+0000 7f09c7ad4640 -1 Falling back to public interface 2024-09-10T19:57:05.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:05.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:05.015 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:57:05.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:57:05.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:05.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:05.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:06.418 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:06.415+0000 7f09c7ad4640 -1 osd.4 0 log_to_monitors true 2024-09-10T19:57:06.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:06.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:06.449 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:57:06.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:57:06.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:06.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:07.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:08.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:08.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:08.148 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:57:08.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:57:08.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:08.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:08.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:09.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:09.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:09.652 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:57:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:57:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:09.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:10.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:11.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:11.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:11.156 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:57:11.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:57:11.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:11.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:11.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:12.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:12.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:12.576 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:57:12.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:57:12.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:12.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:12.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:13.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:13.976 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:13.976 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:57:13.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:57:13.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:13.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:14.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:15.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:15.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:15.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:57:15.446 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:57:15.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:15.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:15.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:16.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:16.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:16.907 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:57:16.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:57:16.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:16.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:17.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:18.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:18.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:18.390 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:57:18.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:57:18.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:18.390 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:18.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:19.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:19.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:19.943 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:57:19.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:57:19.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:19.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:21.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:22.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:22.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:22.141 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:57:22.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:57:22.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:22.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:22.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:23.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:23.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:23.640 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:57:23.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:57:23.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:23.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:24.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:25.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:25.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:25.121 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:57:25.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:57:25.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:25.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:25.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:26.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:26.871 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:57:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:57:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:27.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:28.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:28.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:28.716 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:57:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:57:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:28.717 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:29.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:30.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:30.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:30.663 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:57:30.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:57:30.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:30.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T19:57:31.060 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1814896357,v1:127.0.0.1:6835/1814896357] [v2:127.0.0.1:6836/1814896357,v1:127.0.0.1:6837/1814896357] exists,up fd4c87c8-cbaa-41c2-b8f9-c568f6055f21 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 5 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T19:57:31.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2024-09-10T19:57:31.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:57:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c08a352-cdc9-4245-801b-6873c2afae45 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 1c08a352-cdc9-4245-801b-6873c2afae45' 2024-09-10T19:57:31.067 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 1c08a352-cdc9-4245-801b-6873c2afae45 2024-09-10T19:57:31.068 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:57:31.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCrpOBmwCHcBBAA6X34+r3eA3kWthG+R2+xdg== 2024-09-10T19:57:31.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCrpOBmwCHcBBAA6X34+r3eA3kWthG+R2+xdg=="}' 2024-09-10T19:57:31.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c08a352-cdc9-4245-801b-6873c2afae45 -i td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:57:31.572 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:57:31.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2024-09-10T19:57:31.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrpOBmwCHcBBAA6X34+r3eA3kWthG+R2+xdg== --osd-uuid 1c08a352-cdc9-4245-801b-6873c2afae45 2024-09-10T19:57:31.618 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:31.616+0000 7f7b106b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:31.625 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:31.623+0000 7f7b106b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:31.624+0000 7f7b106b2640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:31.624+0000 7f7b106b2640 -1 bdev(0x55f49a7a3c00 td/osd-backfill-recovery-log/5/block) open stat got: (1) Operation not permitted 2024-09-10T19:57:31.626 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:31.624+0000 7f7b106b2640 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2024-09-10T19:57:38.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2024-09-10T19:57:38.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:57:38.246 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-10T19:57:38.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T19:57:38.247 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:57:38.781 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-10T19:57:38.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T19:57:38.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:57:38.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:57:38.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:57:38.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:57:38.815 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:38.812+0000 7f79d8e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:38.838 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:38.836+0000 7f79d8e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:38.839 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:38.837+0000 7f79d8e2e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:57:39.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T19:57:39.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:57:39.261 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:39.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:39.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:39.680 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:39.678+0000 7f79d8e2e640 -1 Falling back to public interface 2024-09-10T19:57:40.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:40.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:40.679 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:57:40.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:57:40.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:40.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:41.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:41.666 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:57:41.663+0000 7f79d8e2e640 -1 osd.5 0 log_to_monitors true 2024-09-10T19:57:42.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:42.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:42.140 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:57:42.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:57:42.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:42.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:42.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:43.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:43.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:43.704 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:57:43.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:57:43.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:43.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:44.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:45.136 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:45.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:45.137 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:57:45.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:57:45.137 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:45.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:45.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:46.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:46.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:46.608 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:57:46.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:57:46.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:46.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:47.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:48.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:48.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:48.031 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:57:48.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:57:48.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:48.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:48.479 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:49.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:49.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:49.481 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:57:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:57:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:49.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:49.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:50.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:50.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:50.911 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:57:50.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:57:50.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:50.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:51.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:52.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:52.339 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:57:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:57:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:52.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:52.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:53.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:53.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:53.880 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:57:53.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:57:53.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:53.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:54.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:55.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:55.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:55.320 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:57:55.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:57:55.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:55.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:55.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:56.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:56.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:56.854 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:57:56.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:57:56.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:56.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:57.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:57:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:57:58.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:57:58.563 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:57:58.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:57:58.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:57:58.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:57:59.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:00.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:00.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:00.271 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:58:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:58:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:00.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:58:01.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:02.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:02.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:02.304 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:58:02.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:58:02.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:02.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:58:03.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:04.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:04.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:04.086 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:58:04.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:58:04.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:04.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:58:04.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:05.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:05.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:05.851 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:58:05.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:58:05.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:05.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T19:58:06.292 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1595635128,v1:127.0.0.1:6843/1595635128] [v2:127.0.0.1:6844/1595635128,v1:127.0.0.1:6845/1595635128] exists,up 1c08a352-cdc9-4245-801b-6873c2afae45 2024-09-10T19:58:06.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:58:06.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:58:06.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:58:06.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:60: _common_test: create_pool test 1 1 2024-09-10T19:58:06.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T19:58:06.789 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2024-09-10T19:58:06.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T19:58:07.803 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: seq 1 40 2024-09-10T19:58:07.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:07.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-1 /etc/passwd 2024-09-10T19:58:08.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:08.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-2 /etc/passwd 2024-09-10T19:58:08.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:08.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-3 /etc/passwd 2024-09-10T19:58:08.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:08.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-4 /etc/passwd 2024-09-10T19:58:08.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:08.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-5 /etc/passwd 2024-09-10T19:58:08.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:08.904 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-6 /etc/passwd 2024-09-10T19:58:09.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-7 /etc/passwd 2024-09-10T19:58:09.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-8 /etc/passwd 2024-09-10T19:58:09.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-9 /etc/passwd 2024-09-10T19:58:09.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.491 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-10 /etc/passwd 2024-09-10T19:58:09.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-11 /etc/passwd 2024-09-10T19:58:09.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-12 /etc/passwd 2024-09-10T19:58:09.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:09.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-13 /etc/passwd 2024-09-10T19:58:10.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-14 /etc/passwd 2024-09-10T19:58:10.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-15 /etc/passwd 2024-09-10T19:58:10.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-16 /etc/passwd 2024-09-10T19:58:10.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-17 /etc/passwd 2024-09-10T19:58:10.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-18 /etc/passwd 2024-09-10T19:58:10.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:10.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-19 /etc/passwd 2024-09-10T19:58:11.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-20 /etc/passwd 2024-09-10T19:58:11.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-21 /etc/passwd 2024-09-10T19:58:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-22 /etc/passwd 2024-09-10T19:58:11.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-23 /etc/passwd 2024-09-10T19:58:11.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-24 /etc/passwd 2024-09-10T19:58:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-25 /etc/passwd 2024-09-10T19:58:11.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:11.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-26 /etc/passwd 2024-09-10T19:58:12.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-27 /etc/passwd 2024-09-10T19:58:12.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-28 /etc/passwd 2024-09-10T19:58:12.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-29 /etc/passwd 2024-09-10T19:58:12.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-30 /etc/passwd 2024-09-10T19:58:12.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-31 /etc/passwd 2024-09-10T19:58:12.826 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-32 /etc/passwd 2024-09-10T19:58:12.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:12.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-33 /etc/passwd 2024-09-10T19:58:13.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-34 /etc/passwd 2024-09-10T19:58:13.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-35 /etc/passwd 2024-09-10T19:58:13.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-36 /etc/passwd 2024-09-10T19:58:13.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-37 /etc/passwd 2024-09-10T19:58:13.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-38 /etc/passwd 2024-09-10T19:58:13.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:13.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-39 /etc/passwd 2024-09-10T19:58:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T19:58:14.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-40 /etc/passwd 2024-09-10T19:58:14.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:58:14.202 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: jq '.pg_stats[0].up[]' 2024-09-10T19:58:14.565 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:58:14.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph osd out 1 0 2 2024-09-10T19:58:15.168 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2024-09-10T19:58:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:69: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:58:15.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: sleep 1 2024-09-10T19:58:16.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:76: _common_test: wait_for_clean 2024-09-10T19:58:16.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T19:58:16.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T19:58:16.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T19:58:16.181 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T19:58:16.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T19:58:16.182 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T19:58:16.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T19:58:16.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T19:58:16.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:58:16.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:58:16.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:16.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:58:16.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836521 2024-09-10T19:58:16.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836521 2024-09-10T19:58:16.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521' 2024-09-10T19:58:16.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:16.841 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:58:17.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672996 2024-09-10T19:58:17.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672996 2024-09-10T19:58:17.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672996' 2024-09-10T19:58:17.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:17.002 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:58:17.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509475 2024-09-10T19:58:17.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509475 2024-09-10T19:58:17.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509475' 2024-09-10T19:58:17.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:17.150 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:58:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345945 2024-09-10T19:58:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345945 2024-09-10T19:58:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509475 3-85899345945' 2024-09-10T19:58:17.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:17.270 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:58:17.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182419 2024-09-10T19:58:17.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182419 2024-09-10T19:58:17.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509475 3-85899345945 4-107374182419' 2024-09-10T19:58:17.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:17.387 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018886 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018886 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509475 3-85899345945 4-107374182419 5-128849018886' 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836521 2024-09-10T19:58:17.536 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:17.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:58:17.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836521 2024-09-10T19:58:17.538 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:17.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836521 2024-09-10T19:58:17.539 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836521 2024-09-10T19:58:17.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-10T19:58:17.539 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:58:17.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836522 -lt 21474836521 2024-09-10T19:58:17.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:17.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672996 2024-09-10T19:58:17.973 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:17.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:58:17.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672996 2024-09-10T19:58:17.974 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:17.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672996 2024-09-10T19:58:17.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-10T19:58:17.975 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672996 2024-09-10T19:58:17.976 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:58:18.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-10T19:58:18.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:18.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509475 2024-09-10T19:58:18.365 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:18.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:58:18.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509475 2024-09-10T19:58:18.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:18.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509475 2024-09-10T19:58:18.367 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509475 2024-09-10T19:58:18.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-10T19:58:18.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:58:18.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-10T19:58:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:18.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345945 2024-09-10T19:58:18.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:58:18.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345945 2024-09-10T19:58:18.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345945 2024-09-10T19:58:18.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345945' 2024-09-10T19:58:18.768 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345945 2024-09-10T19:58:18.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:58:19.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345945 -lt 85899345945 2024-09-10T19:58:19.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:19.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182419 2024-09-10T19:58:19.182 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:19.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:58:19.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182419 2024-09-10T19:58:19.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182419 2024-09-10T19:58:19.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2024-09-10T19:58:19.185 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182419 2024-09-10T19:58:19.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:58:19.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182419 -lt 107374182419 2024-09-10T19:58:19.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:19.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018886 2024-09-10T19:58:19.604 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:19.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:58:19.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018886 2024-09-10T19:58:19.605 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:19.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018886 2024-09-10T19:58:19.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-09-10T19:58:19.606 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018886 2024-09-10T19:58:19.606 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:58:20.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018886 -lt 128849018886 2024-09-10T19:58:20.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T19:58:20.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:20.074 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:20.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:20.679 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:21.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:58:21.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:21.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:21.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:21.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:58:21.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T19:58:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T19:58:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T19:58:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:58:21.618 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:21.719 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:22.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:58:22.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:22.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:22.203 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:58:22.793 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:58:23.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=20760 2024-09-10T19:58:23.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 20760 '!=' null 2024-09-10T19:58:23.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:58:23.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:58:23.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:58:23.489 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:23.490 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:23.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:58:23.885 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:23.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:23.886 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:58:24.497 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:58:25.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=19355 2024-09-10T19:58:25.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 19355 '!=' null 2024-09-10T19:58:25.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:58:25.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:58:25.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:58:25.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:58:25.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:25.125 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:25.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:58:25.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:25.544 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:25.545 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:26.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:58:26.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:58:26.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:58:26.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:58:26.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=16608 2024-09-10T19:58:26.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 16608 '!=' null 2024-09-10T19:58:26.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:58:26.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:58:26.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:26.747 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:26.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:26.748 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:27.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T19:58:27.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:27.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:27.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:27.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T19:58:27.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T19:58:27.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T19:58:28.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=49062 2024-09-10T19:58:28.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 49062 '!=' null 2024-09-10T19:58:28.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T19:58:28.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T19:58:28.194 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T19:58:28.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T19:58:28.295 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T19:58:28.296 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T19:58:28.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T19:58:28.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T19:58:28.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T19:58:28.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:78: _common_test: flush_pg_stats 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T19:58:29.199 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T19:58:29.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T19:58:29.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:29.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T19:58:29.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836526 2024-09-10T19:58:29.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836526 2024-09-10T19:58:29.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526' 2024-09-10T19:58:29.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:29.765 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T19:58:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673000 2024-09-10T19:58:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673000 2024-09-10T19:58:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949673000' 2024-09-10T19:58:29.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:29.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T19:58:30.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509479 2024-09-10T19:58:30.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509479 2024-09-10T19:58:30.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509479' 2024-09-10T19:58:30.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:30.025 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T19:58:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345950 2024-09-10T19:58:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345950 2024-09-10T19:58:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509479 3-85899345950' 2024-09-10T19:58:30.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:30.139 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T19:58:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182423 2024-09-10T19:58:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182423 2024-09-10T19:58:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509479 3-85899345950 4-107374182423' 2024-09-10T19:58:30.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T19:58:30.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018891 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018891 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949673000 2-64424509479 3-85899345950 4-107374182423 5-128849018891' 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836526 2024-09-10T19:58:30.368 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:30.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T19:58:30.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836526 2024-09-10T19:58:30.369 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:30.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836526 2024-09-10T19:58:30.371 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836526 2024-09-10T19:58:30.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-10T19:58:30.371 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:58:30.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836524 -lt 21474836526 2024-09-10T19:58:30.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T19:58:31.794 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T19:58:31.794 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T19:58:32.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-10T19:58:32.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:32.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673000 2024-09-10T19:58:32.229 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:32.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T19:58:32.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673000 2024-09-10T19:58:32.231 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:32.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673000 2024-09-10T19:58:32.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673000' 2024-09-10T19:58:32.232 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673000 2024-09-10T19:58:32.232 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T19:58:32.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673001 -lt 42949673000 2024-09-10T19:58:32.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:32.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509479 2024-09-10T19:58:32.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:32.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T19:58:32.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509479 2024-09-10T19:58:32.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:32.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509479 2024-09-10T19:58:32.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-09-10T19:58:32.649 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509479 2024-09-10T19:58:32.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T19:58:33.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509480 -lt 64424509479 2024-09-10T19:58:33.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:33.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345950 2024-09-10T19:58:33.081 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:33.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T19:58:33.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345950 2024-09-10T19:58:33.082 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:33.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345950 2024-09-10T19:58:33.083 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345950 2024-09-10T19:58:33.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-09-10T19:58:33.084 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T19:58:33.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345950 -lt 85899345950 2024-09-10T19:58:33.498 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:33.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182423 2024-09-10T19:58:33.499 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:33.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T19:58:33.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182423 2024-09-10T19:58:33.500 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:33.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182423 2024-09-10T19:58:33.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182423' 2024-09-10T19:58:33.501 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182423 2024-09-10T19:58:33.502 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T19:58:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182424 -lt 107374182423 2024-09-10T19:58:33.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T19:58:33.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018891 2024-09-10T19:58:33.900 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T19:58:33.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T19:58:33.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018891 2024-09-10T19:58:33.901 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T19:58:33.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018891 2024-09-10T19:58:33.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-09-10T19:58:33.902 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018891 2024-09-10T19:58:33.903 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T19:58:34.285 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018891 -lt 128849018891 2024-09-10T19:58:34.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: ceph pg dump pgs --format=json 2024-09-10T19:58:34.286 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: jq '.pg_stats[0].up_primary' 2024-09-10T19:58:34.651 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T19:58:34.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: newprimary=4 2024-09-10T19:58:34.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:81: _common_test: kill_daemons 2024-09-10T19:58:34.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:58:34.664 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:58:34.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:58:34.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:58:34.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:58:39.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:58:39.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: ERRORS=0 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: tee td/osd-backfill-recovery-log/result.log 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=4 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T19:58:39.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "34'40", 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "0'0", 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c00128d9:::obj-1:head", 2024-09-10T19:58:41.344 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "33'1", 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4377.0:1", 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:07.837478+0000", 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.345 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d7942fc0:::obj-2:head", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'2", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4380.0:1", 2024-09-10T19:58:41.346 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:08.143988+0000", 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.347 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1dda57ab:::obj-3:head", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'3", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4383.0:1", 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.348 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:08.418397+0000", 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.349 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ba1802d0:::obj-4:head", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'4", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4386.0:1", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:08.609391+0000", 2024-09-10T19:58:41.350 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.351 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d5d4a582:::obj-5:head", 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'5", 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4389.0:1", 2024-09-10T19:58:41.352 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:08.767046+0000", 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.353 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:25cac09a:::obj-6:head", 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'6", 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4392.0:1", 2024-09-10T19:58:41.354 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:08.934340+0000", 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.355 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:b11b7838:::obj-7:head", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'7", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4395.0:1", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.084655+0000", 2024-09-10T19:58:41.356 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.357 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a0f702de:::obj-8:head", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'8", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4398.0:1", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.226977+0000", 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.358 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.359 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1e5c3677:::obj-9:head", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'9", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4401.0:1", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.377098+0000", 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.360 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:86dc045c:::obj-10:head", 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'10", 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.361 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4404.0:1", 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.519019+0000", 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.362 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1e6eca61:::obj-11:head", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'11", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4407.0:1", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.670466+0000", 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.363 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.364 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:18dee445:::obj-12:head", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'12", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4410.0:1", 2024-09-10T19:58:41.365 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.838016+0000", 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.366 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:283c5c90:::obj-13:head", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'13", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4413.0:1", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:09.972129+0000", 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.367 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.368 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cc896bf0:::obj-14:head", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'14", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4416.0:1", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.123078+0000", 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.369 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a581f807:::obj-15:head", 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'15", 2024-09-10T19:58:41.370 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4419.0:1", 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.265130+0000", 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.371 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c902ca7c:::obj-16:head", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'16", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4422.0:1", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.398892+0000", 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.372 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.373 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:655283c8:::obj-17:head", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'17", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4425.0:1", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.559957+0000", 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.374 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.375 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d759d762:::obj-18:head", 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'18", 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4428.0:1", 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.709074+0000", 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.376 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1b934a2d:::obj-19:head", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'19", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4431.0:1", 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.377 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:10.877003+0000", 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.378 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'20", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4434.0:1", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.061236+0000", 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.379 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.380 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ab396690:::obj-21:head", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'21", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4437.0:1", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.205115+0000", 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.381 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:254983db:::obj-22:head", 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'22", 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.382 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4440.0:1", 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.346207+0000", 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.383 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e656299f:::obj-23:head", 2024-09-10T19:58:41.384 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'23", 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4443.0:1", 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.487986+0000", 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.385 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5cdf412b:::obj-24:head", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'24", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4446.0:1", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.647957+0000", 2024-09-10T19:58:41.386 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.387 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cacae143:::obj-25:head", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'25", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4449.0:1", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.798694+0000", 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.388 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.389 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:34b5bff6:::obj-26:head", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'26", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4452.0:1", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:11.965541+0000", 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.390 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a354f8f6:::obj-27:head", 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'27", 2024-09-10T19:58:41.391 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4455.0:1", 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.150819+0000", 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.392 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7421db49:::obj-28:head", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'28", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4458.0:1", 2024-09-10T19:58:41.393 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.293016+0000", 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.394 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:54bc79ba:::obj-29:head", 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'29", 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4461.0:1", 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.395 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.435668+0000", 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.396 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:01b15595:::obj-30:head", 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'30", 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4464.0:1", 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.397 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.578269+0000", 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.398 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'31", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4467.0:1", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.712656+0000", 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.399 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.400 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1bf2562f:::obj-32:head", 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'32", 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4470.0:1", 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.854938+0000", 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.401 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:b982ae22:::obj-33:head", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'33", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4473.0:1", 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.402 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:12.997400+0000", 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.403 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1ad9f98e:::obj-34:head", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'34", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4476.0:1", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.139736+0000", 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.404 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.405 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:941f43ff:::obj-35:head", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'35", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4479.0:1", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.282510+0000", 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.406 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.407 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:24ebabe9:::obj-36:head", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'36", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4482.0:1", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.424604+0000", 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.408 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5f3a5398:::obj-37:head", 2024-09-10T19:58:41.409 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'37", 2024-09-10T19:58:41.605 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.605 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4485.0:1", 2024-09-10T19:58:41.605 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.605 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.567415+0000", 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.606 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c96cb37e:::obj-38:head", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'38", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4488.0:1", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.723411+0000", 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.607 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.608 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:69e36996:::obj-39:head", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'39", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4491.0:1", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:13.917907+0000", 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.609 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:9f75773e:::obj-40:head", 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'40", 2024-09-10T19:58:41.610 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4494.0:1", 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T19:58:14.087673+0000", 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T19:58:41.611 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [] 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2024-09-10T19:58:41.612 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2024-09-10T19:58:41.613 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2024-09-10T19:58:41.613 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T19:58:41.613 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T19:58:41.613 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:58:41.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: LOGLEN=40 2024-09-10T19:58:41.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:86: _common_test: '[' 40 '!=' 40 ']' 2024-09-10T19:58:41.614 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2024-09-10T19:58:41.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: DUPSLEN=0 2024-09-10T19:58:41.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:91: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:58:41.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'40 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'39 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'38 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'37 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'36 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'35 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'34 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'33 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'32 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'31 2024-09-10T19:58:41.620 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'30 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'29 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'28 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'27 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'26 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'25 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'24 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'23 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'22 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'21 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'20 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'19 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'18 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'17 2024-09-10T19:58:41.621 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'16 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'15 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'14 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'13 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'12 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'11 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'10 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'9 2024-09-10T19:58:41.622 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'8 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'7 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'6 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'5 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'4 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'3 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 34'2 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after copy log version 33'1 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T19:58:17.071+0000 7f165c9d3640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.627 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'40 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'39 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'38 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'37 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'36 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'35 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'34 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'33 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'32 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'31 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'30 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'29 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'28 2024-09-10T19:58:41.628 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'27 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'26 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'25 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'24 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'23 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'22 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'21 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'20 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'19 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'18 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'17 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'16 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'15 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'14 2024-09-10T19:58:41.629 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'13 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'12 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'11 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'10 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'9 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'8 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'7 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'6 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'5 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'4 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'3 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'2 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 33'1 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.630 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'40 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'39 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'38 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'37 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'36 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'35 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'34 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'33 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'32 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'31 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'30 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'29 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'28 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'27 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'26 2024-09-10T19:58:41.631 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'25 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'24 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'23 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'22 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'21 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'20 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'19 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'18 2024-09-10T19:58:41.632 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'17 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'16 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'15 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'14 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'13 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'12 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'11 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'10 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'9 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'8 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'7 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'6 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'5 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'4 2024-09-10T19:58:41.633 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'3 2024-09-10T19:58:41.634 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 34'2 2024-09-10T19:58:41.634 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after copy log version 33'1 2024-09-10T19:58:41.634 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.4.log:2024-09-10T19:58:17.347+0000 7f09aba46640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:97: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:35: run: teardown td/osd-backfill-recovery-log 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:58:41.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:58:41.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:58:41.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:58:41.641 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:58:41.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:58:41.642 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:58:41.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:58:41.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:58:41.644 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:58:41.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:58:41.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:58:41.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:58:41.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:58:41.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:58:41.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:58:41.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:58:41.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:58:41.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:41.678 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:41.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:58:41.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:58:41.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:32: run: for func in $funcs 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:33: run: setup td/osd-backfill-recovery-log 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-recovery-log 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T19:58:41.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T19:58:41.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T19:58:41.681 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T19:58:41.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T19:58:41.682 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T19:58:41.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T19:58:41.683 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T19:58:41.683 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T19:58:41.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:58:41.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T19:58:41.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T19:58:41.685 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T19:58:41.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:58:41.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T19:58:41.687 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T19:58:41.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T19:58:41.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:41.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:41.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T19:58:41.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T19:58:41.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T19:58:41.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-recovery-log 2024-09-10T19:58:41.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T19:58:41.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:41.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:41.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.53443 2024-09-10T19:58:41.693 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T19:58:41.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:34: run: TEST_recovery_2 td/osd-backfill-recovery-log 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:130: TEST_recovery_2: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:132: TEST_recovery_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 151 10 141 20 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:41: _common_test: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:42: _common_test: local 'extra_opts=--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:43: _common_test: local loglen=151 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:44: _common_test: local dupslen=10 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:45: _common_test: local objects=141 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:46: _common_test: local moreobjects=20 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:48: _common_test: local OSDS=6 2024-09-10T19:58:41.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:50: _common_test: run_mon td/osd-backfill-recovery-log a 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-recovery-log/a 2024-09-10T19:58:41.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:42.167 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:58:42.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T19:58:42.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T19:58:42.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T19:58:42.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:58:42.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:58:42.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T19:58:42.386 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T19:58:42.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:58:42.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:58:42.387 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:58:42.388 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get fsid 2024-09-10T19:58:42.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T19:58:42.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T19:58:42.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T19:58:42.474 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.53443/ceph-mon.a.asok 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T19:58:42.475 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.53443/ceph-mon.a.asok config get mon_host 2024-09-10T19:58:42.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:51: _common_test: run_mgr td/osd-backfill-recovery-log x 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-recovery-log/x 2024-09-10T19:58:42.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:43.398 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:43.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:58:43.399 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:58:43.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T19:58:43.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:52: _common_test: export CEPH_ARGS 2024-09-10T19:58:43.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:58:43.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:53: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:58:43.431 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: expr 6 - 1 2024-09-10T19:58:43.432 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: seq 0 5 2024-09-10T19:58:43.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:58:43.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 0 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:58:43.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:58:43.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd518cc9-e40c-474c-b8b3-06b5b648097d 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fd518cc9-e40c-474c-b8b3-06b5b648097d' 2024-09-10T19:58:43.439 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 fd518cc9-e40c-474c-b8b3-06b5b648097d 2024-09-10T19:58:43.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:58:43.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDzpOBmbYobGxAA1drvpNJFuYZ60MToxzXX3g== 2024-09-10T19:58:43.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDzpOBmbYobGxAA1drvpNJFuYZ60MToxzXX3g=="}' 2024-09-10T19:58:43.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd518cc9-e40c-474c-b8b3-06b5b648097d -i td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:58:43.725 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:58:43.736 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2024-09-10T19:58:43.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzpOBmbYobGxAA1drvpNJFuYZ60MToxzXX3g== --osd-uuid fd518cc9-e40c-474c-b8b3-06b5b648097d 2024-09-10T19:58:43.772 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:43.768+0000 7f557e8a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:43.794 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:43.791+0000 7f557e8a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:43.795 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:43.792+0000 7f557e8a5640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:43.796 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:43.793+0000 7f557e8a5640 -1 bdev(0x556953a99c00 td/osd-backfill-recovery-log/0/block) open stat got: (1) Operation not permitted 2024-09-10T19:58:43.796 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:43.793+0000 7f557e8a5640 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2024-09-10T19:58:49.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2024-09-10T19:58:49.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:58:49.751 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T19:58:49.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T19:58:49.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:58:50.291 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T19:58:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T19:58:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:58:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:58:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:58:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:58:50.324 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:50.320+0000 7f488c664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:50.332 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:50.328+0000 7f488c664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:50.333 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:50.330+0000 7f488c664640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:58:50.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:50.706 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:58:50.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:58:50.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:50.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:51.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:51.697 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:51.694+0000 7f488c664640 -1 Falling back to public interface 2024-09-10T19:58:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:52.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:52.131 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:58:52.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:58:52.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:52.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:52.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:53.301 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:58:53.299+0000 7f488c664640 -1 osd.0 0 log_to_monitors true 2024-09-10T19:58:53.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:53.558 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:53.558 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:58:53.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:58:53.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:53.559 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:54.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:55.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:55.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:55.036 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:58:55.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:58:55.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:55.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:55.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:56.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:56.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:56.434 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:58:56.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:58:56.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:56.435 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:56.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:57.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:57.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:57.831 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:58:57.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:58:57.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:57.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:58.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:58:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:58:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:58:59.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:58:59.328 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:58:59.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:58:59.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:58:59.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:00.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:00.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:00.723 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:59:00.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:59:00.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:00.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:01.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:02.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:02.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:02.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:59:02.149 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:59:02.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:02.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:02.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:03.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:03.548 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:03.548 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:59:03.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:59:03.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:03.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:04.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:05.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:05.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:05.071 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:59:05.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:59:05.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:05.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:05.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:06.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:06.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:06.982 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:59:06.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:59:06.982 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:06.983 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:07.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:08.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:08.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:08.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:59:08.615 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:59:08.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:08.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:09.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:10.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:10.312 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:10.312 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:59:10.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:59:10.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:10.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:11.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:12.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:12.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:12.089 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:59:12.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:59:12.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:12.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:13.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:14.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:59:14.008 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:59:14.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:14.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:14.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:15.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:15.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:15.702 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:59:15.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:59:15.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:15.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T19:59:16.116 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1695511359,v1:127.0.0.1:6803/1695511359] [v2:127.0.0.1:6804/1695511359,v1:127.0.0.1:6805/1695511359] exists,up fd518cc9-e40c-474c-b8b3-06b5b648097d 2024-09-10T19:59:16.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 1 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:59:16.120 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:59:16.121 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:59:16.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2024-09-10T19:59:16.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:59:16.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1cf65224-f8eb-4955-af1e-351514db5b74 2024-09-10T19:59:16.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 1cf65224-f8eb-4955-af1e-351514db5b74' 2024-09-10T19:59:16.123 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 1cf65224-f8eb-4955-af1e-351514db5b74 2024-09-10T19:59:16.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:59:16.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAUpeBmo4A1CBAATDQbSg6vx85UoYAithYIvg== 2024-09-10T19:59:16.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAUpeBmo4A1CBAATDQbSg6vx85UoYAithYIvg=="}' 2024-09-10T19:59:16.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1cf65224-f8eb-4955-af1e-351514db5b74 -i td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:59:16.758 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:59:16.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2024-09-10T19:59:16.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUpeBmo4A1CBAATDQbSg6vx85UoYAithYIvg== --osd-uuid 1cf65224-f8eb-4955-af1e-351514db5b74 2024-09-10T19:59:16.805 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:16.802+0000 7f6df062a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:16.816 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:16.814+0000 7f6df062a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:16.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:16.815+0000 7f6df062a640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:16.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:16.815+0000 7f6df062a640 -1 bdev(0x5605be6fdc00 td/osd-backfill-recovery-log/1/block) open stat got: (1) Operation not permitted 2024-09-10T19:59:16.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:16.815+0000 7f6df062a640 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2024-09-10T19:59:22.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2024-09-10T19:59:22.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:59:22.830 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T19:59:22.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T19:59:22.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:59:23.367 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T19:59:23.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T19:59:23.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:59:23.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:59:23.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:59:23.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:59:23.400 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:23.398+0000 7f6cf9811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:23.415 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:23.412+0000 7f6cf9811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:23.416 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:23.413+0000 7f6cf9811640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:59:23.846 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:23.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:59:23.847 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:59:23.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:23.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:24.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:25.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:25.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:25.265 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:59:25.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:59:25.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:25.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:25.269 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:25.267+0000 7f6cf9811640 -1 Falling back to public interface 2024-09-10T19:59:25.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:26.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:26.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:26.701 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:59:26.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T19:59:26.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:26.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:27.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:27.168 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:27.166+0000 7f6cf9811640 -1 osd.1 0 log_to_monitors true 2024-09-10T19:59:28.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:28.091 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T19:59:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T19:59:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:28.499 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:29.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:29.501 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T19:59:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T19:59:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:29.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:29.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:30.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:30.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:30.926 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T19:59:30.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T19:59:30.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:30.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:31.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:32.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:32.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:32.328 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T19:59:32.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T19:59:32.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:32.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:32.726 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:33.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:33.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:33.727 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T19:59:33.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T19:59:33.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:33.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:34.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:35.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:35.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:35.168 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T19:59:35.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T19:59:35.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:35.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:35.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:36.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:36.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:36.648 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T19:59:36.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T19:59:36.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:36.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:37.144 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:38.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:38.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:38.146 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T19:59:38.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T19:59:38.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:38.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:38.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:39.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:39.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:39.646 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T19:59:39.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T19:59:39.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:39.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:40.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:41.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:41.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:41.543 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T19:59:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T19:59:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:41.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:42.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:43.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:43.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:43.395 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T19:59:43.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T19:59:43.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:43.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:43.900 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:44.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T19:59:44.902 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T19:59:44.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:44.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:45.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:46.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T19:59:46.608 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T19:59:46.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:46.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:47.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:48.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:48.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:48.686 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T19:59:48.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T19:59:48.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:48.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:49.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:50.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:50.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:50.586 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T19:59:50.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T19:59:50.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:50.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T19:59:51.084 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3108379364,v1:127.0.0.1:6811/3108379364] [v2:127.0.0.1:6812/3108379364,v1:127.0.0.1:6813/3108379364] exists,up 1cf65224-f8eb-4955-af1e-351514db5b74 2024-09-10T19:59:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T19:59:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T19:59:51.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 2 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2024-09-10T19:59:51.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T19:59:51.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T19:59:51.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d86eae92-86ab-4768-918c-d5a6fac533e4 2024-09-10T19:59:51.092 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 d86eae92-86ab-4768-918c-d5a6fac533e4 2024-09-10T19:59:51.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 d86eae92-86ab-4768-918c-d5a6fac533e4' 2024-09-10T19:59:51.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T19:59:51.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3peBmLjBTBhAAsvjuNwqqvx+aoAtfe57a7g== 2024-09-10T19:59:51.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3peBmLjBTBhAAsvjuNwqqvx+aoAtfe57a7g=="}' 2024-09-10T19:59:51.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d86eae92-86ab-4768-918c-d5a6fac533e4 -i td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:59:51.602 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T19:59:51.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2024-09-10T19:59:51.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3peBmLjBTBhAAsvjuNwqqvx+aoAtfe57a7g== --osd-uuid d86eae92-86ab-4768-918c-d5a6fac533e4 2024-09-10T19:59:51.649 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:51.646+0000 7f667e902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:51.652 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:51.650+0000 7f667e902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:51.653 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:51.651+0000 7f667e902640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:51.653 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:51.651+0000 7f667e902640 -1 bdev(0x55edb4129c00 td/osd-backfill-recovery-log/2/block) open stat got: (1) Operation not permitted 2024-09-10T19:59:51.654 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:51.652+0000 7f667e902640 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2024-09-10T19:59:57.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2024-09-10T19:59:57.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T19:59:57.617 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T19:59:57.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T19:59:57.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T19:59:58.152 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T19:59:58.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T19:59:58.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T19:59:58.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T19:59:58.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T19:59:58.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T19:59:58.185 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:58.183+0000 7fa9d11bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:58.198 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:58.195+0000 7fa9d11bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:58.198 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T19:59:58.196+0000 7fa9d11bb640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T19:59:58.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T19:59:58.546 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T19:59:58.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:58.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T19:59:58.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T19:59:59.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T19:59:59.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T19:59:59.934 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T19:59:59.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T19:59:59.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T19:59:59.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:00.045 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:00.043+0000 7fa9d11bb640 -1 Falling back to public interface 2024-09-10T20:00:00.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:01.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:01.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:01.329 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:00:01.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:00:01.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:01.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:01.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:01.832+0000 7fa9d11bb640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:00:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:02.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:02.810 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:00:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:00:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:02.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:03.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:04.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:04.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:04.243 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:00:04.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:00:04.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:04.243 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:04.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:05.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:05.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:05.691 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:00:05.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:00:05.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:05.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:06.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:07.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:07.187 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:07.188 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:00:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:00:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:07.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:07.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:08.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:08.754 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:00:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:00:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:08.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:09.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:10.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:10.175 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:10.176 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:00:10.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:00:10.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:10.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:10.666 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:11.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:11.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:11.668 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:00:11.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:00:11.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:11.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:12.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:13.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:13.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:13.211 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:00:13.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:00:13.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:13.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:13.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:14.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:14.645 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:00:14.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:00:14.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:14.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:15.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:16.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:16.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:16.330 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:00:16.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:00:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:16.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:17.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:18.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:18.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:18.142 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:00:18.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:00:18.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:18.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:19.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:20.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:20.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:20.355 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:00:20.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:00:20.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:20.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:21.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:22.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:22.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:22.083 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:00:22.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:00:22.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:22.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:23.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:24.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:24.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:24.230 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:00:24.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:00:24.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:24.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:00:24.639 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3113974948,v1:127.0.0.1:6819/3113974948] [v2:127.0.0.1:6820/3113974948,v1:127.0.0.1:6821/3113974948] exists,up d86eae92-86ab-4768-918c-d5a6fac533e4 2024-09-10T20:00:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:00:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:00:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:00:24.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 3 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2024-09-10T20:00:24.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:00:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e 2024-09-10T20:00:24.646 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e' 2024-09-10T20:00:24.647 INFO:tasks.workunit.client.0.smithi130.stdout:add osd3 e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e 2024-09-10T20:00:24.647 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:00:24.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBYpeBmqL5nJxAAqWuWoZ7fiZNZ0B3yNtVkHg== 2024-09-10T20:00:24.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBYpeBmqL5nJxAAqWuWoZ7fiZNZ0B3yNtVkHg=="}' 2024-09-10T20:00:24.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e -i td/osd-backfill-recovery-log/3/new.json 2024-09-10T20:00:25.208 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:00:25.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2024-09-10T20:00:25.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYpeBmqL5nJxAAqWuWoZ7fiZNZ0B3yNtVkHg== --osd-uuid e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e 2024-09-10T20:00:25.254 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:25.252+0000 7f5d21a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:25.256 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:25.254+0000 7f5d21a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:25.257 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:25.255+0000 7f5d21a2f640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:25.258 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:25.255+0000 7f5d21a2f640 -1 bdev(0x55ea4ecedc00 td/osd-backfill-recovery-log/3/block) open stat got: (1) Operation not permitted 2024-09-10T20:00:25.258 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:25.256+0000 7f5d21a2f640 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2024-09-10T20:00:31.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2024-09-10T20:00:31.281 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:00:31.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2024-09-10T20:00:31.282 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd3 key to auth repository 2024-09-10T20:00:31.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:00:31.816 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.3 2024-09-10T20:00:31.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2024-09-10T20:00:31.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:00:31.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:00:31.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:00:31.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:00:31.849 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:31.845+0000 7f5bed60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:31.851 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:31.847+0000 7f5bed60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:31.852 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:31.848+0000 7f5bed60e640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:32.218 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:00:32.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:00:32.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:32.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:32.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:32.700 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:32.697+0000 7f5bed60e640 -1 Falling back to public interface 2024-09-10T20:00:33.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:33.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:33.639 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:00:33.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:00:33.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:33.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:34.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:34.651 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:34.647+0000 7f5bed60e640 -1 osd.3 0 log_to_monitors true 2024-09-10T20:00:35.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:35.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:35.103 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:00:35.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:00:35.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:35.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:35.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:36.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:36.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:36.587 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:00:36.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:00:36.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:36.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:37.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:38.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:38.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:38.012 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:00:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:00:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:38.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:38.549 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:39.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:39.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:39.551 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:00:39.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:00:39.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:39.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:39.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:40.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:40.990 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:00:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:00:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:40.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:41.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:42.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:42.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:42.530 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:00:42.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:00:42.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:42.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:42.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:43.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:43.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:43.958 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:00:43.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:00:43.958 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:43.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:44.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:45.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:45.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:45.357 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:00:45.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:00:45.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:45.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:45.967 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:46.968 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:46.969 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:00:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:00:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:46.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:47.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:48.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:48.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:48.587 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:00:48.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:00:48.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:48.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:49.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:50.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:50.292 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:00:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:00:50.292 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:50.293 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:51.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:52.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:52.355 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:00:52.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:00:52.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:52.355 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:53.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:54.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:54.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:54.166 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:00:54.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:00:54.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:54.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:54.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:55.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:55.910 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:00:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:00:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:55.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:56.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:00:57.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:00:57.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:00:57.822 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:00:57.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:00:57.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:00:57.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2024-09-10T20:00:58.267 INFO:tasks.workunit.client.0.smithi130.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1483865053,v1:127.0.0.1:6827/1483865053] [v2:127.0.0.1:6828/1483865053,v1:127.0.0.1:6829/1483865053] exists,up e0e5a4eb-7daa-4c33-bc9b-a1e279f2949e 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 4 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:00:58.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T20:00:58.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:00:58.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2024-09-10T20:00:58.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:00:58.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=cc3ab6ac-16a1-4e4b-b9ff-60cc86077024 2024-09-10T20:00:58.274 INFO:tasks.workunit.client.0.smithi130.stdout:add osd4 cc3ab6ac-16a1-4e4b-b9ff-60cc86077024 2024-09-10T20:00:58.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 cc3ab6ac-16a1-4e4b-b9ff-60cc86077024' 2024-09-10T20:00:58.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:00:58.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6peBmtfQxERAAosSKkr6nzld9IM/QlT6BGQ== 2024-09-10T20:00:58.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6peBmtfQxERAAosSKkr6nzld9IM/QlT6BGQ=="}' 2024-09-10T20:00:58.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new cc3ab6ac-16a1-4e4b-b9ff-60cc86077024 -i td/osd-backfill-recovery-log/4/new.json 2024-09-10T20:00:58.849 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:00:58.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2024-09-10T20:00:58.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6peBmtfQxERAAosSKkr6nzld9IM/QlT6BGQ== --osd-uuid cc3ab6ac-16a1-4e4b-b9ff-60cc86077024 2024-09-10T20:00:58.897 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:58.895+0000 7fe98ae30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:58.907 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:58.905+0000 7fe98ae30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:58.908 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:58.906+0000 7fe98ae30640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:00:58.909 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:58.906+0000 7fe98ae30640 -1 bdev(0x563c3d311c00 td/osd-backfill-recovery-log/4/block) open stat got: (1) Operation not permitted 2024-09-10T20:00:58.909 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:00:58.906+0000 7fe98ae30640 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2024-09-10T20:01:04.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2024-09-10T20:01:04.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:01:04.935 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd4 key to auth repository 2024-09-10T20:01:04.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2024-09-10T20:01:04.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:01:05.475 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.4 2024-09-10T20:01:05.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2024-09-10T20:01:05.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:01:05.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:01:05.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:01:05.476 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:01:05.508 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:05.506+0000 7f39b87cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:05.518 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:05.516+0000 7f39b87cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:05.519 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:05.517+0000 7f39b87cc640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:05.943 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:01:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:01:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:05.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:06.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:07.110 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:07.108+0000 7f39b87cc640 -1 Falling back to public interface 2024-09-10T20:01:07.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:07.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:07.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:01:07.368 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:01:07.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:07.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:07.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:08.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:08.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:08.758 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:01:08.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:01:08.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:08.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:08.940 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:08.938+0000 7f39b87cc640 -1 osd.4 0 log_to_monitors true 2024-09-10T20:01:09.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:10.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:10.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:10.158 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:01:10.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:01:10.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:10.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:10.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:11.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:11.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:11.556 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:01:11.556 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:01:11.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:11.557 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:11.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:12.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:12.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:12.956 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:01:12.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:01:12.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:12.956 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:13.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:14.382 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:01:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:01:14.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:14.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:14.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:15.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:15.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:15.913 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:01:15.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:01:15.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:15.913 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:16.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:17.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:17.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:17.310 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:01:17.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:01:17.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:17.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:17.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:18.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:18.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:18.797 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:01:18.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:01:18.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:18.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:19.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:20.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:20.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:20.191 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:01:20.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:01:20.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:20.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:20.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:21.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:21.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:21.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:01:21.591 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:01:21.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:21.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:22.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:23.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:23.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:23.330 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:01:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:01:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:23.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:23.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:24.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:24.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:24.784 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:01:24.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:01:24.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:24.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:25.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:26.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:01:26.413 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:01:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:27.195 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:28.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:28.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:28.197 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:01:28.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:01:28.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:28.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:28.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:29.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:29.853 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:01:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:01:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:29.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:30.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:31.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:31.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:31.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:01:31.545 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T20:01:31.545 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:31.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2024-09-10T20:01:32.031 INFO:tasks.workunit.client.0.smithi130.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3238680759,v1:127.0.0.1:6835/3238680759] [v2:127.0.0.1:6836/3238680759,v1:127.0.0.1:6837/3238680759] exists,up cc3ab6ac-16a1-4e4b-b9ff-60cc86077024 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:55: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:57: _common_test: run_osd td/osd-backfill-recovery-log 5 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-recovery-log 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2024-09-10T20:01:32.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:01:32.036 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:01:32.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d28458e4-4ad1-4553-8d5c-e6d67647f448 2024-09-10T20:01:32.038 INFO:tasks.workunit.client.0.smithi130.stdout:add osd5 d28458e4-4ad1-4553-8d5c-e6d67647f448 2024-09-10T20:01:32.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 d28458e4-4ad1-4553-8d5c-e6d67647f448' 2024-09-10T20:01:32.039 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:01:32.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCcpeBmOtoiAxAAXIvmwE5NXYYqlEaFhtO5yQ== 2024-09-10T20:01:32.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCcpeBmOtoiAxAAXIvmwE5NXYYqlEaFhtO5yQ=="}' 2024-09-10T20:01:32.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d28458e4-4ad1-4553-8d5c-e6d67647f448 -i td/osd-backfill-recovery-log/5/new.json 2024-09-10T20:01:32.507 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:01:32.518 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2024-09-10T20:01:32.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCcpeBmOtoiAxAAXIvmwE5NXYYqlEaFhtO5yQ== --osd-uuid d28458e4-4ad1-4553-8d5c-e6d67647f448 2024-09-10T20:01:32.554 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:32.552+0000 7fddec2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:32.582 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:32.580+0000 7fddec2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:32.583 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:32.581+0000 7fddec2f1640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:32.583 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:32.581+0000 7fddec2f1640 -1 bdev(0x557a2f292800 td/osd-backfill-recovery-log/5/block) open stat got: (1) Operation not permitted 2024-09-10T20:01:32.583 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:32.581+0000 7fddec2f1640 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2024-09-10T20:01:39.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2024-09-10T20:01:39.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:01:39.064 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd5 key to auth repository 2024-09-10T20:01:39.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2024-09-10T20:01:39.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:01:39.637 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.5 2024-09-10T20:01:39.637 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2024-09-10T20:01:39.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=99980513-10a5-4d30-8174-16af2d9974ea --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.53443/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:01:39.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:01:39.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:01:39.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:01:39.671 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:39.669+0000 7f1c74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:39.673 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:39.671+0000 7f1c74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:39.674 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:39.672+0000 7f1c74a15640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:01:40.086 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:01:40.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:40.087 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:40.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:41.019 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:41.017+0000 7f1c74a15640 -1 Falling back to public interface 2024-09-10T20:01:41.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:41.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:41.513 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:01:41.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:01:41.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:41.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:41.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:42.625 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:01:42.623+0000 7f1c74a15640 -1 osd.5 0 log_to_monitors true 2024-09-10T20:01:42.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:42.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:42.928 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:01:42.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:01:42.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:42.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:43.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:44.411 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:44.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:44.412 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:01:44.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:01:44.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:44.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:44.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:45.843 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:01:45.843 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:01:45.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:45.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:46.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:47.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:47.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:47.383 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:01:47.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:01:47.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:47.383 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:47.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:48.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:48.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:48.907 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:01:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:01:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:49.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:50.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:50.422 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:01:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:01:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:50.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:50.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:51.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:51.844 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:01:51.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:01:51.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:51.845 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:52.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:53.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:53.332 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:01:53.332 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:53.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:01:53.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:53.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:53.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:54.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:54.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:54.732 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:01:54.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:01:54.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:54.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:55.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:56.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:56.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:56.131 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:01:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:01:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:56.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:57.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:57.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:57.820 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:01:57.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:01:57.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:57.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:01:58.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:01:59.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:01:59.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:01:59.709 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:01:59.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:01:59.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:01:59.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:02:00.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:01.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:01.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:01.836 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:02:01.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:02:01.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:01.836 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:02:02.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:03.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:03.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:03.941 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:02:03.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:02:03.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:03.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:02:04.753 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:05.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:05.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:05.755 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:02:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:02:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:05.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:02:06.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:02:07.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:02:07.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:02:07.572 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T20:02:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:02:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:02:07.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2024-09-10T20:02:08.059 INFO:tasks.workunit.client.0.smithi130.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3620295958,v1:127.0.0.1:6843/3620295958] [v2:127.0.0.1:6844/3620295958,v1:127.0.0.1:6845/3620295958] exists,up d28458e4-4ad1-4553-8d5c-e6d67647f448 2024-09-10T20:02:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:02:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:02:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:02:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:60: _common_test: create_pool test 1 1 2024-09-10T20:02:08.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:02:08.696 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2024-09-10T20:02:08.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:02:09.709 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: seq 1 141 2024-09-10T20:02:09.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:09.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-1 /etc/passwd 2024-09-10T20:02:10.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-2 /etc/passwd 2024-09-10T20:02:10.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-3 /etc/passwd 2024-09-10T20:02:10.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-4 /etc/passwd 2024-09-10T20:02:10.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-5 /etc/passwd 2024-09-10T20:02:10.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-6 /etc/passwd 2024-09-10T20:02:10.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:10.922 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-7 /etc/passwd 2024-09-10T20:02:11.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-8 /etc/passwd 2024-09-10T20:02:11.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-9 /etc/passwd 2024-09-10T20:02:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-10 /etc/passwd 2024-09-10T20:02:11.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-11 /etc/passwd 2024-09-10T20:02:11.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.735 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-12 /etc/passwd 2024-09-10T20:02:11.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:11.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-13 /etc/passwd 2024-09-10T20:02:12.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.045 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-14 /etc/passwd 2024-09-10T20:02:12.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-15 /etc/passwd 2024-09-10T20:02:12.429 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-16 /etc/passwd 2024-09-10T20:02:12.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-17 /etc/passwd 2024-09-10T20:02:12.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-18 /etc/passwd 2024-09-10T20:02:12.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:12.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-19 /etc/passwd 2024-09-10T20:02:13.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-20 /etc/passwd 2024-09-10T20:02:13.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-21 /etc/passwd 2024-09-10T20:02:13.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-22 /etc/passwd 2024-09-10T20:02:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-23 /etc/passwd 2024-09-10T20:02:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-24 /etc/passwd 2024-09-10T20:02:13.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-25 /etc/passwd 2024-09-10T20:02:13.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:13.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-26 /etc/passwd 2024-09-10T20:02:14.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-27 /etc/passwd 2024-09-10T20:02:14.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-28 /etc/passwd 2024-09-10T20:02:14.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-29 /etc/passwd 2024-09-10T20:02:14.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-30 /etc/passwd 2024-09-10T20:02:14.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-31 /etc/passwd 2024-09-10T20:02:14.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:14.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-32 /etc/passwd 2024-09-10T20:02:15.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.051 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-33 /etc/passwd 2024-09-10T20:02:15.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-34 /etc/passwd 2024-09-10T20:02:15.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-35 /etc/passwd 2024-09-10T20:02:15.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-36 /etc/passwd 2024-09-10T20:02:15.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-37 /etc/passwd 2024-09-10T20:02:15.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:15.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-38 /etc/passwd 2024-09-10T20:02:16.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-39 /etc/passwd 2024-09-10T20:02:16.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.157 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-40 /etc/passwd 2024-09-10T20:02:16.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-41 /etc/passwd 2024-09-10T20:02:16.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-42 /etc/passwd 2024-09-10T20:02:16.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-43 /etc/passwd 2024-09-10T20:02:16.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-44 /etc/passwd 2024-09-10T20:02:16.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:16.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-45 /etc/passwd 2024-09-10T20:02:17.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-46 /etc/passwd 2024-09-10T20:02:17.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-47 /etc/passwd 2024-09-10T20:02:17.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-48 /etc/passwd 2024-09-10T20:02:17.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-49 /etc/passwd 2024-09-10T20:02:17.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-50 /etc/passwd 2024-09-10T20:02:17.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:17.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-51 /etc/passwd 2024-09-10T20:02:18.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-52 /etc/passwd 2024-09-10T20:02:18.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-53 /etc/passwd 2024-09-10T20:02:18.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-54 /etc/passwd 2024-09-10T20:02:18.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.553 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-55 /etc/passwd 2024-09-10T20:02:18.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-56 /etc/passwd 2024-09-10T20:02:18.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.854 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-57 /etc/passwd 2024-09-10T20:02:18.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:18.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-58 /etc/passwd 2024-09-10T20:02:19.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-59 /etc/passwd 2024-09-10T20:02:19.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-60 /etc/passwd 2024-09-10T20:02:19.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-61 /etc/passwd 2024-09-10T20:02:19.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-62 /etc/passwd 2024-09-10T20:02:19.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-63 /etc/passwd 2024-09-10T20:02:19.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:19.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-64 /etc/passwd 2024-09-10T20:02:20.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.128 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-65 /etc/passwd 2024-09-10T20:02:20.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-66 /etc/passwd 2024-09-10T20:02:20.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-67 /etc/passwd 2024-09-10T20:02:20.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-68 /etc/passwd 2024-09-10T20:02:20.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.756 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-69 /etc/passwd 2024-09-10T20:02:20.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:20.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-70 /etc/passwd 2024-09-10T20:02:21.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-71 /etc/passwd 2024-09-10T20:02:21.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-72 /etc/passwd 2024-09-10T20:02:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-73 /etc/passwd 2024-09-10T20:02:21.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-74 /etc/passwd 2024-09-10T20:02:21.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-75 /etc/passwd 2024-09-10T20:02:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.853 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-76 /etc/passwd 2024-09-10T20:02:21.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:21.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-77 /etc/passwd 2024-09-10T20:02:22.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:22.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-78 /etc/passwd 2024-09-10T20:02:22.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:22.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-79 /etc/passwd 2024-09-10T20:02:22.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:22.682 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-80 /etc/passwd 2024-09-10T20:02:22.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:22.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-81 /etc/passwd 2024-09-10T20:02:23.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-82 /etc/passwd 2024-09-10T20:02:23.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-83 /etc/passwd 2024-09-10T20:02:23.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-84 /etc/passwd 2024-09-10T20:02:23.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.470 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-85 /etc/passwd 2024-09-10T20:02:23.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-86 /etc/passwd 2024-09-10T20:02:23.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-87 /etc/passwd 2024-09-10T20:02:23.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:23.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-88 /etc/passwd 2024-09-10T20:02:24.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-89 /etc/passwd 2024-09-10T20:02:24.273 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-90 /etc/passwd 2024-09-10T20:02:24.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-91 /etc/passwd 2024-09-10T20:02:24.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-92 /etc/passwd 2024-09-10T20:02:24.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-93 /etc/passwd 2024-09-10T20:02:24.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:24.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-94 /etc/passwd 2024-09-10T20:02:25.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-95 /etc/passwd 2024-09-10T20:02:25.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-96 /etc/passwd 2024-09-10T20:02:25.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.354 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-97 /etc/passwd 2024-09-10T20:02:25.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-98 /etc/passwd 2024-09-10T20:02:25.722 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.723 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-99 /etc/passwd 2024-09-10T20:02:25.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:25.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-100 /etc/passwd 2024-09-10T20:02:26.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-101 /etc/passwd 2024-09-10T20:02:26.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-102 /etc/passwd 2024-09-10T20:02:26.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-103 /etc/passwd 2024-09-10T20:02:26.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.501 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-104 /etc/passwd 2024-09-10T20:02:26.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-105 /etc/passwd 2024-09-10T20:02:26.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-106 /etc/passwd 2024-09-10T20:02:26.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:26.953 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-107 /etc/passwd 2024-09-10T20:02:27.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-108 /etc/passwd 2024-09-10T20:02:27.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-109 /etc/passwd 2024-09-10T20:02:27.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-110 /etc/passwd 2024-09-10T20:02:27.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-111 /etc/passwd 2024-09-10T20:02:27.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.792 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-112 /etc/passwd 2024-09-10T20:02:27.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:27.951 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-113 /etc/passwd 2024-09-10T20:02:28.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-114 /etc/passwd 2024-09-10T20:02:28.302 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-115 /etc/passwd 2024-09-10T20:02:28.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.453 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-116 /etc/passwd 2024-09-10T20:02:28.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-117 /etc/passwd 2024-09-10T20:02:28.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-118 /etc/passwd 2024-09-10T20:02:28.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:28.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-119 /etc/passwd 2024-09-10T20:02:29.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-120 /etc/passwd 2024-09-10T20:02:29.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-121 /etc/passwd 2024-09-10T20:02:29.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-122 /etc/passwd 2024-09-10T20:02:29.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.512 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-123 /etc/passwd 2024-09-10T20:02:29.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-124 /etc/passwd 2024-09-10T20:02:29.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:29.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-125 /etc/passwd 2024-09-10T20:02:30.002 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-126 /etc/passwd 2024-09-10T20:02:30.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-127 /etc/passwd 2024-09-10T20:02:30.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-128 /etc/passwd 2024-09-10T20:02:30.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-129 /etc/passwd 2024-09-10T20:02:30.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-130 /etc/passwd 2024-09-10T20:02:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-131 /etc/passwd 2024-09-10T20:02:30.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:30.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-132 /etc/passwd 2024-09-10T20:02:31.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-133 /etc/passwd 2024-09-10T20:02:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-134 /etc/passwd 2024-09-10T20:02:31.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-135 /etc/passwd 2024-09-10T20:02:31.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-136 /etc/passwd 2024-09-10T20:02:31.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-137 /etc/passwd 2024-09-10T20:02:31.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:31.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-138 /etc/passwd 2024-09-10T20:02:32.071 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:32.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-139 /etc/passwd 2024-09-10T20:02:32.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:32.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-140 /etc/passwd 2024-09-10T20:02:32.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:62: _common_test: for j in $(seq 1 $objects) 2024-09-10T20:02:32.375 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:64: _common_test: rados -p test put obj-141 /etc/passwd 2024-09-10T20:02:32.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph pg dump pgs --format=json 2024-09-10T20:02:32.527 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: jq '.pg_stats[0].up[]' 2024-09-10T20:02:32.895 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T20:02:32.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:68: _common_test: ceph osd out 1 0 2 2024-09-10T20:02:33.398 INFO:tasks.workunit.client.0.smithi130.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2024-09-10T20:02:33.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:69: _common_test: '[' 20 '!=' 0 ']' 2024-09-10T20:02:33.409 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: seq 1 20 2024-09-10T20:02:33.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:33.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-1 /etc/passwd 2024-09-10T20:02:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:35.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-2 /etc/passwd 2024-09-10T20:02:35.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:35.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-3 /etc/passwd 2024-09-10T20:02:36.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:36.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-4 /etc/passwd 2024-09-10T20:02:36.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:36.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-5 /etc/passwd 2024-09-10T20:02:36.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:36.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-6 /etc/passwd 2024-09-10T20:02:36.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:36.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-7 /etc/passwd 2024-09-10T20:02:37.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:37.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-8 /etc/passwd 2024-09-10T20:02:37.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:37.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-9 /etc/passwd 2024-09-10T20:02:37.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:37.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-10 /etc/passwd 2024-09-10T20:02:38.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:38.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-11 /etc/passwd 2024-09-10T20:02:38.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:38.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-12 /etc/passwd 2024-09-10T20:02:38.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:38.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-13 /etc/passwd 2024-09-10T20:02:38.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:38.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-14 /etc/passwd 2024-09-10T20:02:39.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:39.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-15 /etc/passwd 2024-09-10T20:02:39.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:39.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-16 /etc/passwd 2024-09-10T20:02:39.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:39.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-17 /etc/passwd 2024-09-10T20:02:39.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:39.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-18 /etc/passwd 2024-09-10T20:02:40.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:40.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-19 /etc/passwd 2024-09-10T20:02:40.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:70: _common_test: for j in $(seq 1 $moreobjects) 2024-09-10T20:02:40.317 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:72: _common_test: rados -p test put obj-more-20 /etc/passwd 2024-09-10T20:02:40.560 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:75: _common_test: sleep 1 2024-09-10T20:02:41.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:76: _common_test: wait_for_clean 2024-09-10T20:02:41.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:02:41.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:02:41.562 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:02:41.562 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:02:41.563 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:02:41.563 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:02:41.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:02:41.563 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:02:41.564 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:02:41.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:02:41.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:02:41.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:02:41.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:02:41.661 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:02:41.662 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.054 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:02:42.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836526 2024-09-10T20:02:42.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836526 2024-09-10T20:02:42.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526' 2024-09-10T20:02:42.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.174 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:02:42.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672999 2024-09-10T20:02:42.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672999 2024-09-10T20:02:42.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672999' 2024-09-10T20:02:42.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:02:42.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509478 2024-09-10T20:02:42.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509478 2024-09-10T20:02:42.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672999 2-64424509478' 2024-09-10T20:02:42.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.508 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:02:42.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345950 2024-09-10T20:02:42.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345950 2024-09-10T20:02:42.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672999 2-64424509478 3-85899345950' 2024-09-10T20:02:42.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.624 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T20:02:42.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182420 2024-09-10T20:02:42.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182420 2024-09-10T20:02:42.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672999 2-64424509478 3-85899345950 4-107374182420' 2024-09-10T20:02:42.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:02:42.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T20:02:42.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018892 2024-09-10T20:02:42.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018892 2024-09-10T20:02:42.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836526 1-42949672999 2-64424509478 3-85899345950 4-107374182420 5-128849018892' 2024-09-10T20:02:42.857 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:42.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836526 2024-09-10T20:02:42.858 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:42.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:02:42.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836526 2024-09-10T20:02:42.859 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:42.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836526 2024-09-10T20:02:42.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-10T20:02:42.860 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836526 2024-09-10T20:02:42.860 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:02:43.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836523 -lt 21474836526 2024-09-10T20:02:43.252 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:02:44.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:02:44.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:02:44.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-10T20:02:44.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:44.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672999 2024-09-10T20:02:44.648 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:44.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:02:44.649 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672999 2024-09-10T20:02:44.650 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:44.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672999 2024-09-10T20:02:44.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-09-10T20:02:44.651 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672999 2024-09-10T20:02:44.651 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:02:45.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672999 -lt 42949672999 2024-09-10T20:02:45.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:45.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509478 2024-09-10T20:02:45.153 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:45.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:02:45.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509478 2024-09-10T20:02:45.154 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:45.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509478 2024-09-10T20:02:45.155 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509478 2024-09-10T20:02:45.156 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-10T20:02:45.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:02:45.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509479 -lt 64424509478 2024-09-10T20:02:45.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:45.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345950 2024-09-10T20:02:45.700 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:45.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:02:45.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345950 2024-09-10T20:02:45.701 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:45.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345950 2024-09-10T20:02:45.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-09-10T20:02:45.702 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345950 2024-09-10T20:02:45.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:02:46.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345951 -lt 85899345950 2024-09-10T20:02:46.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:46.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182420 2024-09-10T20:02:46.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:46.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T20:02:46.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182420 2024-09-10T20:02:46.239 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:46.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182420 2024-09-10T20:02:46.240 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182420 2024-09-10T20:02:46.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182420' 2024-09-10T20:02:46.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T20:02:46.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182420 -lt 107374182420 2024-09-10T20:02:46.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:02:46.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018892 2024-09-10T20:02:46.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:02:46.767 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T20:02:46.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018892 2024-09-10T20:02:46.768 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:02:46.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018892 2024-09-10T20:02:46.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018892' 2024-09-10T20:02:46.769 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018892 2024-09-10T20:02:46.770 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T20:02:47.271 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018892 -lt 128849018892 2024-09-10T20:02:47.271 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:02:47.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:47.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:47.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:02:47.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:47.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:47.946 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:47.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:47.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:47.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:47.947 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:48.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:48.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:48.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:48.330 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:49.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:49.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:02:49.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:02:49.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:02:49.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:49.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:49.123 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:49.546 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:49.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:49.546 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:49.547 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:50.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:50.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:50.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:50.240 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:50.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=70265 2024-09-10T20:02:50.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 70265 '!=' null 2024-09-10T20:02:50.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:50.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:50.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:50.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:50.979 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:50.979 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:50.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:50.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:50.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:50.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:50.980 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:51.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:51.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:51.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:51.449 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:52.130 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:52.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=70266 2024-09-10T20:02:52.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 70266 '!=' null 2024-09-10T20:02:52.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:52.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:52.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:52.844 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:52.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:52.845 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:53.284 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:53.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:53.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:53.284 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:54.110 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=113916 2024-09-10T20:02:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 113916 '!=' null 2024-09-10T20:02:54.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:54.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:54.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:54.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:54.848 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:54.849 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:55.303 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:55.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:55.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:55.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:55.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:55.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:55.851 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:56.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=81977 2024-09-10T20:02:56.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 81977 '!=' null 2024-09-10T20:02:56.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:56.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:56.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:56.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:56.643 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:57.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:57.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:57.058 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:57.059 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:57.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:57.586 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:58.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=65477 2024-09-10T20:02:58.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 65477 '!=' null 2024-09-10T20:02:58.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:58.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:58.143 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:58.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:58.244 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:58.244 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:58.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:58.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:58.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:58.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:58.245 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:02:58.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:02:58.619 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:02:58.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:02:58.620 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:02:59.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:02:59.142 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:02:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=111789 2024-09-10T20:02:59.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 111789 '!=' null 2024-09-10T20:02:59.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:02:59.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:02:59.678 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:02:59.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:02:59.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:02:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:02:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:02:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:02:59.779 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:02:59.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:02:59.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:03:00.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:03:00.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:03:00.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:03:00.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:78: _common_test: flush_pg_stats 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:03:00.680 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:2 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:3 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:4 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:5' 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.063 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:03:01.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836531 2024-09-10T20:03:01.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836531 2024-09-10T20:03:01.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531' 2024-09-10T20:03:01.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:03:01.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673005 2024-09-10T20:03:01.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673005 2024-09-10T20:03:01.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-42949673005' 2024-09-10T20:03:01.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.297 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:03:01.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509484 2024-09-10T20:03:01.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509484 2024-09-10T20:03:01.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-42949673005 2-64424509484' 2024-09-10T20:03:01.412 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.412 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-10T20:03:01.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=85899345956 2024-09-10T20:03:01.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 85899345956 2024-09-10T20:03:01.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-42949673005 2-64424509484 3-85899345956' 2024-09-10T20:03:01.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-10T20:03:01.644 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=107374182425 2024-09-10T20:03:01.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 107374182425 2024-09-10T20:03:01.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-42949673005 2-64424509484 3-85899345956 4-107374182425' 2024-09-10T20:03:01.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:03:01.645 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-10T20:03:01.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=128849018897 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 128849018897 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836531 1-42949673005 2-64424509484 3-85899345956 4-107374182425 5-128849018897' 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836531 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:01.760 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:03:01.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836531 2024-09-10T20:03:01.761 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:01.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836531 2024-09-10T20:03:01.762 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836531 2024-09-10T20:03:01.762 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836531' 2024-09-10T20:03:01.763 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:03:02.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836531 -lt 21474836531 2024-09-10T20:03:02.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:02.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673005 2024-09-10T20:03:02.183 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:02.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:03:02.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673005 2024-09-10T20:03:02.184 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:02.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673005 2024-09-10T20:03:02.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-09-10T20:03:02.185 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673005 2024-09-10T20:03:02.185 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:03:02.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673005 -lt 42949673005 2024-09-10T20:03:02.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:02.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509484 2024-09-10T20:03:02.609 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:02.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:03:02.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509484 2024-09-10T20:03:02.610 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:02.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509484 2024-09-10T20:03:02.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-09-10T20:03:02.611 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509484 2024-09-10T20:03:02.611 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:03:03.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509482 -lt 64424509484 2024-09-10T20:03:03.034 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:03:04.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:03:04.036 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:03:04.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509484 -lt 64424509484 2024-09-10T20:03:04.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:04.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 3-85899345956 2024-09-10T20:03:04.455 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:04.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=3 2024-09-10T20:03:04.456 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 3-85899345956 2024-09-10T20:03:04.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:04.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=85899345956 2024-09-10T20:03:04.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.3 seq 85899345956' 2024-09-10T20:03:04.458 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.3 seq 85899345956 2024-09-10T20:03:04.458 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-10T20:03:04.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 85899345957 -lt 85899345956 2024-09-10T20:03:04.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:04.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 4-107374182425 2024-09-10T20:03:04.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:04.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=4 2024-09-10T20:03:04.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 4-107374182425 2024-09-10T20:03:04.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:04.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=107374182425 2024-09-10T20:03:04.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-09-10T20:03:04.883 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.4 seq 107374182425 2024-09-10T20:03:04.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-10T20:03:05.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 107374182426 -lt 107374182425 2024-09-10T20:03:05.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:03:05.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 5-128849018897 2024-09-10T20:03:05.298 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:03:05.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=5 2024-09-10T20:03:05.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 5-128849018897 2024-09-10T20:03:05.299 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:03:05.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=128849018897 2024-09-10T20:03:05.300 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.5 seq 128849018897 2024-09-10T20:03:05.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-09-10T20:03:05.301 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-10T20:03:05.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 128849018898 -lt 128849018897 2024-09-10T20:03:05.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: ceph pg dump pgs --format=json 2024-09-10T20:03:05.766 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: jq '.pg_stats[0].up_primary' 2024-09-10T20:03:06.135 INFO:tasks.workunit.client.0.smithi130.stderr:dumped pgs 2024-09-10T20:03:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:80: _common_test: newprimary=4 2024-09-10T20:03:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:81: _common_test: kill_daemons 2024-09-10T20:03:06.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:06.148 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:06.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:06.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:06.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:11.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:83: _common_test: ERRORS=0 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:84: _common_test: tee td/osd-backfill-recovery-log/result.log 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=4 2024-09-10T20:03:11.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2024-09-10T20:03:11.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2024-09-10T20:03:11.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2024-09-10T20:03:13.054 INFO:tasks.workunit.client.0.smithi130.stdout:{ 2024-09-10T20:03:13.054 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_log_t": { 2024-09-10T20:03:13.054 INFO:tasks.workunit.client.0.smithi130.stdout: "head": "38'161", 2024-09-10T20:03:13.054 INFO:tasks.workunit.client.0.smithi130.stdout: "tail": "34'10", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "log": [ 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1e6eca61:::obj-11:head", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'11", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4405.0:1", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:11.604455+0000", 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.055 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.056 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:18dee445:::obj-12:head", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'12", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4408.0:1", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:11.763540+0000", 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.057 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.058 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.059 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:283c5c90:::obj-13:head", 2024-09-10T20:03:13.060 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'13", 2024-09-10T20:03:13.060 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.060 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4411.0:1", 2024-09-10T20:03:13.060 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.060 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:11.914611+0000", 2024-09-10T20:03:13.061 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.061 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.061 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.061 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.061 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.062 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cc896bf0:::obj-14:head", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'14", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4414.0:1", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.073545+0000", 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.063 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.064 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a581f807:::obj-15:head", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'15", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4417.0:1", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.299188+0000", 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.065 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.066 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c902ca7c:::obj-16:head", 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'16", 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4420.0:1", 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.067 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.457614+0000", 2024-09-10T20:03:13.068 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.068 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.068 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.068 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.068 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.069 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:655283c8:::obj-17:head", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'17", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4423.0:1", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.600269+0000", 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.070 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.071 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d759d762:::obj-18:head", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'18", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4426.0:1", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.776272+0000", 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.072 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.073 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.074 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1b934a2d:::obj-19:head", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'19", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4429.0:1", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:12.943811+0000", 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.075 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.076 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.077 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'20", 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4432.0:1", 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.094498+0000", 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.078 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.079 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ab396690:::obj-21:head", 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'21", 2024-09-10T20:03:13.080 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4435.0:1", 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.244690+0000", 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.081 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.082 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:254983db:::obj-22:head", 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'22", 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4438.0:1", 2024-09-10T20:03:13.083 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.395630+0000", 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.084 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.085 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e656299f:::obj-23:head", 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'23", 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4441.0:1", 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.086 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.563414+0000", 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.087 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5cdf412b:::obj-24:head", 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'24", 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4444.0:1", 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.088 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.714551+0000", 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.089 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cacae143:::obj-25:head", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'25", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4447.0:1", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:13.865368+0000", 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.090 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:34b5bff6:::obj-26:head", 2024-09-10T20:03:13.091 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'26", 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4450.0:1", 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.015716+0000", 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.092 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a354f8f6:::obj-27:head", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'27", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4453.0:1", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.158102+0000", 2024-09-10T20:03:13.093 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.094 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7421db49:::obj-28:head", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'28", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4456.0:1", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.317688+0000", 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.095 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:54bc79ba:::obj-29:head", 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'29", 2024-09-10T20:03:13.096 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4459.0:1", 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.468135+0000", 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.097 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:01b15595:::obj-30:head", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'30", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4462.0:1", 2024-09-10T20:03:13.098 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.627148+0000", 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.099 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'31", 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4465.0:1", 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.100 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.778046+0000", 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.101 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1bf2562f:::obj-32:head", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'32", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4468.0:1", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:14.929230+0000", 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.102 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.103 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:b982ae22:::obj-33:head", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'33", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4471.0:1", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.079385+0000", 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.104 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.105 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1ad9f98e:::obj-34:head", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'34", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4474.0:1", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.239505+0000", 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.106 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.107 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:941f43ff:::obj-35:head", 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'35", 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4477.0:1", 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.389378+0000", 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.108 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:24ebabe9:::obj-36:head", 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'36", 2024-09-10T20:03:13.109 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4480.0:1", 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.570883+0000", 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.110 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5f3a5398:::obj-37:head", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'37", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4483.0:1", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.742468+0000", 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.111 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.112 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c96cb37e:::obj-38:head", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'38", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4486.0:1", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:15.883935+0000", 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.113 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.114 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:69e36996:::obj-39:head", 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'39", 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4489.0:1", 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.115 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.034880+0000", 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.116 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:9f75773e:::obj-40:head", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'40", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4492.0:1", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.185670+0000", 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.117 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.118 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2c01cd48:::obj-41:head", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'41", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4495.0:1", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.337082+0000", 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.119 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.120 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5e676e3d:::obj-42:head", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'42", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4498.0:1", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.512122+0000", 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.121 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.122 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1215f697:::obj-43:head", 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'43", 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4501.0:1", 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.662840+0000", 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.123 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3a0aa357:::obj-44:head", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'44", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4504.0:1", 2024-09-10T20:03:13.124 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.821518+0000", 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.125 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6366950a:::obj-45:head", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'45", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4507.0:1", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:16.972539+0000", 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.126 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.127 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:621cb2b3:::obj-46:head", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'46", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4510.0:1", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.123793+0000", 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.128 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.129 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d83ab3cd:::obj-47:head", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'47", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4513.0:1", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.273933+0000", 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.130 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3d8e6d4a:::obj-48:head", 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'48", 2024-09-10T20:03:13.131 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4516.0:1", 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.455319+0000", 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.132 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:50b1e2f2:::obj-49:head", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'49", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4519.0:1", 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.133 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.666624+0000", 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.134 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c75430e6:::obj-50:head", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'50", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4522.0:1", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.819019+0000", 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.135 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.136 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:eaf3ca0a:::obj-51:head", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'51", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4525.0:1", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:17.970338+0000", 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.137 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:86a15bea:::obj-52:head", 2024-09-10T20:03:13.138 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'52", 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4528.0:1", 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.119978+0000", 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.139 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1df15dae:::obj-53:head", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'53", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4531.0:1", 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.140 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.279658+0000", 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.141 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:29304dc8:::obj-54:head", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'54", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4534.0:1", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.430803+0000", 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.142 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:de02c26d:::obj-55:head", 2024-09-10T20:03:13.143 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'55", 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4537.0:1", 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.580804+0000", 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.144 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0d36fc08:::obj-56:head", 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'56", 2024-09-10T20:03:13.145 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4540.0:1", 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.731379+0000", 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.146 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2b2cbcba:::obj-57:head", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'57", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4543.0:1", 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.147 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:18.882176+0000", 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.148 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:838be76e:::obj-58:head", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'58", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4546.0:1", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.024511+0000", 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.149 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.150 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2c66fd6f:::obj-59:head", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'59", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4549.0:1", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.200575+0000", 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.151 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2b18273b:::obj-60:head", 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'60", 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.152 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4552.0:1", 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.351737+0000", 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.153 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:fcfa3cd1:::obj-61:head", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'61", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4555.0:1", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.511108+0000", 2024-09-10T20:03:13.154 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.155 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:16e51a68:::obj-62:head", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'62", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4558.0:1", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.694291+0000", 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.156 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c15678fc:::obj-63:head", 2024-09-10T20:03:13.157 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'63", 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4561.0:1", 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.845722+0000", 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.158 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e066252d:::obj-64:head", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'64", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4564.0:1", 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.159 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:19.996908+0000", 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.160 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:95a4ceaf:::obj-65:head", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'65", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4567.0:1", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.157496+0000", 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.161 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.162 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6e97b31a:::obj-66:head", 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'66", 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4570.0:1", 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.307321+0000", 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.163 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ceed8b4e:::obj-67:head", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'67", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4573.0:1", 2024-09-10T20:03:13.164 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.467046+0000", 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.165 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:05e2e722:::obj-68:head", 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'68", 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4576.0:1", 2024-09-10T20:03:13.166 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.626008+0000", 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.167 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c40f7de9:::obj-69:head", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'69", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4579.0:1", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.784870+0000", 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.168 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.169 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7bab3581:::obj-70:head", 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'70", 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4582.0:1", 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:20.936453+0000", 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.170 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cb9edc4f:::obj-71:head", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'71", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4585.0:1", 2024-09-10T20:03:13.171 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.086601+0000", 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.172 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:88ae729f:::obj-72:head", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'72", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4588.0:1", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.253694+0000", 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.173 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.174 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:887b5de4:::obj-73:head", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'73", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4591.0:1", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.404222+0000", 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.175 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:996efeaa:::obj-74:head", 2024-09-10T20:03:13.176 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'74", 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4594.0:1", 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.554525+0000", 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.177 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:942222a0:::obj-75:head", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'75", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4597.0:1", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.714821+0000", 2024-09-10T20:03:13.178 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.179 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:35f0499f:::obj-76:head", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'76", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4600.0:1", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:21.881212+0000", 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.180 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:15cd38e3:::obj-77:head", 2024-09-10T20:03:13.181 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'77", 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4603.0:1", 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:22.023665+0000", 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.182 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7adeb77b:::obj-78:head", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'78", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4606.0:1", 2024-09-10T20:03:13.183 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:22.183592+0000", 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.184 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c92176ea:::obj-79:head", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'79", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4609.0:1", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:22.342395+0000", 2024-09-10T20:03:13.185 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.186 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:14abac33:::obj-80:head", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'80", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4612.0:1", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:22.711896+0000", 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.187 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ea1005df:::obj-81:head", 2024-09-10T20:03:13.188 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'81", 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4615.0:1", 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:22.862060+0000", 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.189 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:df18954d:::obj-82:head", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'82", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4618.0:1", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.046156+0000", 2024-09-10T20:03:13.190 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.191 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:cdd3834b:::obj-83:head", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'83", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4621.0:1", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.198113+0000", 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.192 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:dfbf520a:::obj-84:head", 2024-09-10T20:03:13.193 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'84", 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4624.0:1", 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.349128+0000", 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.194 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1dd3df2f:::obj-85:head", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'85", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4627.0:1", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.498765+0000", 2024-09-10T20:03:13.195 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.196 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:54554b77:::obj-86:head", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'86", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4630.0:1", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.672533+0000", 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.197 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:43a48f98:::obj-87:head", 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'87", 2024-09-10T20:03:13.198 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4633.0:1", 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:23.833603+0000", 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.199 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3f7b3ecc:::obj-88:head", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'88", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4636.0:1", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.000743+0000", 2024-09-10T20:03:13.200 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.201 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d25ef7d2:::obj-89:head", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'89", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4639.0:1", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.159957+0000", 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.202 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.203 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ea27d688:::obj-90:head", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'90", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4642.0:1", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.301892+0000", 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.204 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:088f9d12:::obj-91:head", 2024-09-10T20:03:13.205 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'91", 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4645.0:1", 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.460742+0000", 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.206 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.207 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:206f25a5:::obj-92:head", 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'92", 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4648.0:1", 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.611526+0000", 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.208 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:501cca61:::obj-93:head", 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'93", 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.209 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4651.0:1", 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.762599+0000", 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.210 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1ed468c0:::obj-94:head", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'94", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4654.0:1", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:24.921345+0000", 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.211 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.212 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:e8cba918:::obj-95:head", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'95", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4657.0:1", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.072108+0000", 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.213 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.214 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:403c3dfc:::obj-96:head", 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'96", 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4660.0:1", 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.231498+0000", 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.215 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7c2d0cb1:::obj-97:head", 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'97", 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.216 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4663.0:1", 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.382075+0000", 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.217 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6f3e26a7:::obj-98:head", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'98", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4666.0:1", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.580385+0000", 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.218 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.219 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a4b22c6a:::obj-99:head", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'99", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4669.0:1", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.752074+0000", 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.220 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:25c09ea8:::obj-100:head", 2024-09-10T20:03:13.221 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'100", 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4672.0:1", 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:25.901626+0000", 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.222 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:be8b0193:::obj-101:head", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'101", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4675.0:1", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.052520+0000", 2024-09-10T20:03:13.223 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.224 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6aa5e219:::obj-102:head", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'102", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4678.0:1", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.228238+0000", 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.225 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.226 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ff8cd32f:::obj-103:head", 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'103", 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4681.0:1", 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.378880+0000", 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.227 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:fcaf5c1c:::obj-104:head", 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'104", 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.228 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4684.0:1", 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.529003+0000", 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.229 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2111aa60:::obj-105:head", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'105", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4687.0:1", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.688697+0000", 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.230 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.231 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:9fed4eb7:::obj-106:head", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'106", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4690.0:1", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.839636+0000", 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.232 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0b0c0bce:::obj-107:head", 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'107", 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.233 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4693.0:1", 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:26.981324+0000", 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.234 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:b7ce613a:::obj-108:head", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'108", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4696.0:1", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.140716+0000", 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.235 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.236 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:4f32cd61:::obj-109:head", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'109", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4699.0:1", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.291765+0000", 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.237 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:40f97f88:::obj-110:head", 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'110", 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.238 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4702.0:1", 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.456659+0000", 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.239 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:1d8e7a62:::obj-111:head", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'111", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4705.0:1", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.667896+0000", 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.240 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.241 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3a283967:::obj-112:head", 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'112", 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4708.0:1", 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.819817+0000", 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.242 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:2acdc9e7:::obj-113:head", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'113", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4711.0:1", 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.243 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:27.979064+0000", 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.244 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7f18a9ad:::obj-114:head", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'114", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4714.0:1", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.179840+0000", 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.245 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.246 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:852b8a34:::obj-115:head", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'115", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4717.0:1", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.330500+0000", 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.247 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:589de4f7:::obj-116:head", 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'116", 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.248 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4720.0:1", 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.481262+0000", 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.249 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:14f2571c:::obj-117:head", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'117", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4723.0:1", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.640207+0000", 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.250 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.251 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:dd36f0e5:::obj-118:head", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'118", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4726.0:1", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.782638+0000", 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.252 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.253 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0d7fa432:::obj-119:head", 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'119", 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4729.0:1", 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:28.925412+0000", 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.254 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.255 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:295ec31a:::obj-120:head", 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'120", 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4732.0:1", 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.076359+0000", 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.256 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c2863561:::obj-121:head", 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'121", 2024-09-10T20:03:13.257 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4735.0:1", 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.226886+0000", 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.258 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0b943493:::obj-122:head", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'122", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4738.0:1", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.378123+0000", 2024-09-10T20:03:13.259 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.260 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:4cfecbf8:::obj-123:head", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'123", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4741.0:1", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.558940+0000", 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.261 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:fd18e511:::obj-124:head", 2024-09-10T20:03:13.262 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'124", 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4744.0:1", 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.721582+0000", 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.263 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:4f299322:::obj-125:head", 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'125", 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.264 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4747.0:1", 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:29.880352+0000", 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.265 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:302ecc7e:::obj-126:head", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'126", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4750.0:1", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.030694+0000", 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.266 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.267 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:42feb657:::obj-127:head", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'127", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4753.0:1", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.189883+0000", 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.268 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:5e6b9a81:::obj-128:head", 2024-09-10T20:03:13.269 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'128", 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4756.0:1", 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.349521+0000", 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.270 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0d242242:::obj-129:head", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'129", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4759.0:1", 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.271 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.500305+0000", 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.272 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ed409294:::obj-130:head", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'130", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4762.0:1", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.659404+0000", 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.273 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.274 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:720761ff:::obj-131:head", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'131", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4765.0:1", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.810079+0000", 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.275 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.276 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7715ee72:::obj-132:head", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'132", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4768.0:1", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:30.977111+0000", 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.277 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:580938e3:::obj-133:head", 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'133", 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.278 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4771.0:1", 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.127769+0000", 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.279 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:05dc98b5:::obj-134:head", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'134", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4774.0:1", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.278853+0000", 2024-09-10T20:03:13.280 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.281 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:73221e51:::obj-135:head", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'135", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4777.0:1", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.450418+0000", 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.282 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3a97ede2:::obj-136:head", 2024-09-10T20:03:13.283 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'136", 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4780.0:1", 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.654728+0000", 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.284 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:9e95a205:::obj-137:head", 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'137", 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.285 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4783.0:1", 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.806982+0000", 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.286 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:27f846bd:::obj-138:head", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'138", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4786.0:1", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:31.949327+0000", 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.287 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.288 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:bb2f7605:::obj-139:head", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'139", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4789.0:1", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:32.099813+0000", 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.289 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:27668d5f:::obj-140:head", 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'140", 2024-09-10T20:03:13.290 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4792.0:1", 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:32.252691+0000", 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.291 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0d70cb86:::obj-141:head", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'141", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4795.0:1", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:32.403693+0000", 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.292 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.293 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:347b39b5:::obj-more-1:head", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'142", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4802.0:1", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:33.488143+0000", 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.294 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:0c63cc63:::obj-more-2:head", 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'143", 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.295 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4805.0:1", 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:35.640848+0000", 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.296 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:3ce69b57:::obj-more-3:head", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'144", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4808.0:1", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:35.856314+0000", 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.297 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:50476200:::obj-more-4:head", 2024-09-10T20:03:13.298 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'145", 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4811.0:1", 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:36.128990+0000", 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.299 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:7a39cfd8:::obj-more-5:head", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'146", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4814.0:1", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:36.389444+0000", 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.300 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.301 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:a7159f95:::obj-more-6:head", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'147", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4817.0:1", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:36.650197+0000", 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.302 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:37f29f5f:::obj-more-7:head", 2024-09-10T20:03:13.303 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'148", 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4820.0:1", 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:36.943402+0000", 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.304 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:4e4836c1:::obj-more-8:head", 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'149", 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.305 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4823.0:1", 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:37.209485+0000", 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.306 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:d56cad57:::obj-more-9:head", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'150", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4826.0:1", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:37.497146+0000", 2024-09-10T20:03:13.307 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.308 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:96ff4b77:::obj-more-10:head", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'151", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4829.0:1", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:37.852970+0000", 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.309 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.310 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:b8937ca5:::obj-more-11:head", 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'152", 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4832.0:1", 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:38.117228+0000", 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.311 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:c949902e:::obj-more-12:head", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'153", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4835.0:1", 2024-09-10T20:03:13.312 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:38.411614+0000", 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.313 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:4909985f:::obj-more-13:head", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'154", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4838.0:1", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:38.645979+0000", 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.314 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.315 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:acf906ce:::obj-more-14:head", 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'155", 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4841.0:1", 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:38.880145+0000", 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.316 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:44d31ae5:::obj-more-15:head", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'156", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4844.0:1", 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.317 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:39.132646+0000", 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.318 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:54dfd3e0:::obj-more-16:head", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'157", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4847.0:1", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:39.358190+0000", 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.319 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6a7dcd50:::obj-more-17:head", 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'158", 2024-09-10T20:03:13.320 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4850.0:1", 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:39.637243+0000", 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.321 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:fb20876f:::obj-more-18:head", 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'159", 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.322 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4853.0:1", 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:39.852771+0000", 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.323 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.324 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:ed63ae36:::obj-more-19:head", 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'160", 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4856.0:1", 2024-09-10T20:03:13.325 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:40.087439+0000", 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.326 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "op": "modify", 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "object": "1:6a8a392a:::obj-more-20:head", 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "38'161", 2024-09-10T20:03:13.327 INFO:tasks.workunit.client.0.smithi130.stdout: "prior_version": "0'0", 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4859.0:1", 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "extra_reqids": [], 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "mtime": "2024-09-10T20:02:40.349427+0000", 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": 0, 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "mod_desc": { 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "object_mod_desc": { 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "can_local_rollback": false, 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "rollback_info_completed": false, 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: "ops": [] 2024-09-10T20:03:13.328 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_regions": { 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "object_clean_regions": { 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_offsets": "[12777~18446744073709538838]", 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "clean_omap": true, 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "new_object": false 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: ], 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: "dups": [ 2024-09-10T20:03:13.329 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4375.0:1", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'1", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "1", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4378.0:1", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'2", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "2", 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.330 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4381.0:1", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'3", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "3", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4384.0:1", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'4", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "4", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4387.0:1", 2024-09-10T20:03:13.331 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'5", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "5", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4390.0:1", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'6", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "6", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4393.0:1", 2024-09-10T20:03:13.332 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'7", 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "7", 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4396.0:1", 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'8", 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "8", 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.333 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4399.0:1", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'9", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "9", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: { 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "reqid": "client.4402.0:1", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "version": "34'10", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "user_version": "10", 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: "return_code": "0" 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: ] 2024-09-10T20:03:13.334 INFO:tasks.workunit.client.0.smithi130.stdout: }, 2024-09-10T20:03:13.335 INFO:tasks.workunit.client.0.smithi130.stdout: "pg_missing_t": { 2024-09-10T20:03:13.335 INFO:tasks.workunit.client.0.smithi130.stdout: "missing": [], 2024-09-10T20:03:13.335 INFO:tasks.workunit.client.0.smithi130.stdout: "may_include_deletes": true 2024-09-10T20:03:13.335 INFO:tasks.workunit.client.0.smithi130.stdout: } 2024-09-10T20:03:13.335 INFO:tasks.workunit.client.0.smithi130.stdout:} 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:85: _common_test: LOGLEN=151 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:86: _common_test: '[' 151 '!=' 151 ']' 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:90: _common_test: DUPSLEN=10 2024-09-10T20:03:13.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:91: _common_test: '[' 10 '!=' 10 ']' 2024-09-10T20:03:13.337 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:95: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2024-09-10T20:03:13.337 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'141 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'140 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'139 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'138 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'137 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'136 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'135 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'134 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'133 2024-09-10T20:03:13.338 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'132 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'131 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'130 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'129 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'128 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'127 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'126 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'125 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'124 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'123 2024-09-10T20:03:13.339 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'122 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'121 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'120 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'119 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'118 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'117 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'116 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'115 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'114 2024-09-10T20:03:13.340 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'113 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'112 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'111 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'110 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'109 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'108 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'107 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'106 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'105 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'104 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'103 2024-09-10T20:03:13.341 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'102 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'101 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'100 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'99 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'98 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'97 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'96 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'95 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'94 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'93 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'92 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'91 2024-09-10T20:03:13.342 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'90 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'89 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'88 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'87 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'86 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'85 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'84 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'83 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'82 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'81 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'80 2024-09-10T20:03:13.343 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'79 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'78 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'77 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'76 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'75 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'74 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'73 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'72 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'71 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'70 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'69 2024-09-10T20:03:13.344 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'68 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'67 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'66 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'65 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'64 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'63 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'62 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'61 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'60 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'59 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'58 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'57 2024-09-10T20:03:13.345 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'56 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'55 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'54 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'53 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'52 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'51 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'50 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'49 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'48 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'47 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'46 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'45 2024-09-10T20:03:13.346 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'44 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'43 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'42 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'41 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'40 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'39 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'38 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'37 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'36 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'35 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'34 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'33 2024-09-10T20:03:13.347 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'32 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'31 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'30 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'29 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'28 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'27 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'26 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'25 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'24 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'23 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'22 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'21 2024-09-10T20:03:13.348 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'20 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'19 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'18 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'17 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'16 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'15 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'14 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'13 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'12 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'11 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'10 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'9 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'8 2024-09-10T20:03:13.349 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'7 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'6 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'5 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'4 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'3 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'2 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'1 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'141 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'140 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'139 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'138 2024-09-10T20:03:13.350 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'137 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'136 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'135 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'134 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'133 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'132 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'131 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'130 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.765+0000 7f48725da640 20 copy_after copy log version 34'129 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'128 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'127 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'126 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'125 2024-09-10T20:03:13.351 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'124 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'123 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'122 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'121 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'120 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'119 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'118 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'117 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'116 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'115 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'114 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'113 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'112 2024-09-10T20:03:13.352 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'111 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'110 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'109 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'108 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'107 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'106 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'105 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'104 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'103 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'102 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'101 2024-09-10T20:03:13.353 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'100 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'99 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'98 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'97 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'96 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'95 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'94 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'93 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'92 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'91 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'90 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'89 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'88 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'87 2024-09-10T20:03:13.354 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'86 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'85 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'84 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'83 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'82 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'81 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'80 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'79 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'78 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'77 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'76 2024-09-10T20:03:13.355 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'75 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'74 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'73 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'72 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'71 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'70 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'69 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'68 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'67 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'66 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'65 2024-09-10T20:03:13.356 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'64 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'63 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'62 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'61 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'60 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'59 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'58 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'57 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'56 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'55 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'54 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'53 2024-09-10T20:03:13.357 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'52 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'51 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'50 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'49 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'48 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'47 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'46 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'45 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'44 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'43 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'42 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'41 2024-09-10T20:03:13.358 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'40 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'39 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'38 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'37 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'36 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'35 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'34 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'33 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'32 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'31 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'30 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'29 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'28 2024-09-10T20:03:13.359 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'27 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'26 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'25 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'24 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'23 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'22 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'21 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'20 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'19 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'18 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'17 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'16 2024-09-10T20:03:13.360 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'15 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'14 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'13 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'12 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'11 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'10 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'9 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'8 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'7 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'6 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'5 2024-09-10T20:03:13.361 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'4 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'3 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'2 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'1 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'141 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'140 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'139 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'138 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'137 2024-09-10T20:03:13.362 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'136 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'135 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'134 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'133 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'132 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'131 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'130 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'129 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'128 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'127 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'126 2024-09-10T20:03:13.363 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'125 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'124 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'123 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'122 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'121 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'120 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'119 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'118 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'117 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'116 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'115 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'114 2024-09-10T20:03:13.364 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'113 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'112 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'111 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'110 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'109 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'108 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'107 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'106 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'105 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'104 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'103 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'102 2024-09-10T20:03:13.365 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'101 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'100 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'99 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'98 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'97 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'96 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'95 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'94 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'93 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'92 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'91 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'90 2024-09-10T20:03:13.366 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'89 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'88 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'87 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'86 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'85 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'84 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'83 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'82 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'81 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'80 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'79 2024-09-10T20:03:13.367 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'78 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'77 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'76 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'75 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'74 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'73 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'72 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'71 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'70 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'69 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'68 2024-09-10T20:03:13.368 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'67 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'66 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'65 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'64 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'63 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'62 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'61 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'60 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'59 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'58 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'57 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'56 2024-09-10T20:03:13.369 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'55 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'54 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'53 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'52 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'51 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'50 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'49 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'48 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'47 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'46 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'45 2024-09-10T20:03:13.370 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'44 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'43 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'42 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'41 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'40 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'39 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'38 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'37 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'36 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'35 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'34 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'33 2024-09-10T20:03:13.371 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'32 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'31 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'30 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'29 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'28 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'27 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'26 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'25 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'24 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'23 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'22 2024-09-10T20:03:13.372 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'21 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'20 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'19 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'18 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'17 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'16 2024-09-10T20:03:13.373 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'15 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'14 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'13 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'12 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'11 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'10 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'9 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'8 2024-09-10T20:03:13.374 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'7 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'6 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'5 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'4 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'3 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'2 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after copy log version 34'1 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:34.766+0000 7f48725da640 20 copy_after END v 0'0 dups.size()=0 other.dups.size()=0 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_after v 38'161 dups.size()=0 other.dups.size()=10 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'1 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'2 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'3 2024-09-10T20:03:13.375 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'4 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'5 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'6 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'7 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'8 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'9 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup version 34'10 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'11 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'12 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'13 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'14 2024-09-10T20:03:13.376 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'15 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'16 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'17 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'18 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'19 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'20 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'21 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'22 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'23 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'24 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'25 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'26 2024-09-10T20:03:13.377 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'27 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'28 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'29 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'30 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'31 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'32 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'33 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'34 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'35 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'36 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'37 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'38 2024-09-10T20:03:13.378 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'39 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'40 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'41 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'42 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'43 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'44 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'45 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'46 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'47 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'48 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'49 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'50 2024-09-10T20:03:13.379 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'51 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'52 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'53 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'54 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'55 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'56 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'57 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'58 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'59 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'60 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'61 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'62 2024-09-10T20:03:13.380 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'63 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'64 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'65 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'66 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'67 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'68 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'69 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'70 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'71 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'72 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'73 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'74 2024-09-10T20:03:13.381 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'75 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'76 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'77 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'78 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'79 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'80 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'81 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'82 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'83 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'84 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'85 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'86 2024-09-10T20:03:13.382 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'87 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'88 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'89 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'90 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'91 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'92 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'93 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'94 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'95 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'96 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'97 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'98 2024-09-10T20:03:13.383 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'99 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'100 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'101 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'102 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'103 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'104 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'105 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'106 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'107 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'108 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'109 2024-09-10T20:03:13.384 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'110 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'111 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'112 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'113 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'114 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'115 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'116 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'117 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'118 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'119 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'120 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'121 2024-09-10T20:03:13.385 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'122 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'123 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'124 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'125 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'126 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'127 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'128 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'129 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'130 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'131 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'132 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'133 2024-09-10T20:03:13.386 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'134 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'135 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'136 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'137 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'138 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'139 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'140 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 34'141 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'142 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'143 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'144 2024-09-10T20:03:13.387 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'145 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'146 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'147 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'148 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'149 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'150 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'151 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'152 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'153 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'154 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'155 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'156 2024-09-10T20:03:13.388 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'157 2024-09-10T20:03:13.389 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'158 2024-09-10T20:03:13.389 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'159 2024-09-10T20:03:13.389 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'160 2024-09-10T20:03:13.389 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_up_to/copy_after copy dup from log version 38'161 2024-09-10T20:03:13.389 INFO:tasks.workunit.client.0.smithi130.stdout:td/osd-backfill-recovery-log/osd.0.log:2024-09-10T20:02:56.028+0000 7f48705d6640 20 copy_after END v 38'161 dups.size()=161 other.dups.size()=10 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:96: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:97: _common_test: '[' 0 '!=' 0 ']' 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-recovery-log.sh:35: run: teardown td/osd-backfill-recovery-log 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:13.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:13.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:03:13.394 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T20:03:13.414 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:13.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:13.415 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T20:03:13.415 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T20:03:13.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: main: code=0 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-backfill-recovery-log 0 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-recovery-log 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:13.417 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:13.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:13.418 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:13.419 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:13.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:13.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:13.420 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:13.420 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:13.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:13.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:13.422 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:13.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-10T20:03:13.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-recovery-log 2024-09-10T20:03:13.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:13.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:13.425 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.53443 2024-09-10T20:03:13.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.53443 2024-09-10T20:03:13.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:13.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:13.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 0 2024-09-10T20:03:13.427 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:03:13.427 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:03:13.453 INFO:tasks.workunit:Running workunit osd-backfill/osd-backfill-space.sh... 2024-09-10T20:03:13.454 DEBUG:teuthology.orchestra.run.smithi130:workunit test osd-backfill/osd-backfill-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh 2024-09-10T20:03:13.515 INFO:tasks.workunit.client.0.smithi130.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: export PYTHONWARNINGS=ignore 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: main: PYTHONWARNINGS=ignore 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: export CEPH_CONF=/dev/null 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: main: CEPH_CONF=/dev/null 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: main: unset CEPH_ARGS 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: main: local code 2024-09-10T20:03:13.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: main: run td/osd-backfill-space 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:21: run: local dir=td/osd-backfill-space 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:22: run: shift 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:24: run: export CEPH_MON=127.0.0.1:7180 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:24: run: CEPH_MON=127.0.0.1:7180 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:25: run: export CEPH_ARGS 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:26: run: uuidgen 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:26: run: CEPH_ARGS+='--fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none ' 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7180 ' 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:28: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:29: run: CEPH_ARGS+='--fake_statfs_for_testing=3686400 ' 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:30: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-09-10T20:03:13.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:31: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:32: run: export objects=600 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:32: run: objects=600 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:33: run: export poolprefix=test 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:33: run: poolprefix=test 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:35: run: set 2024-09-10T20:03:13.525 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:35: run: local 'funcs=TEST_backfill_grow 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_multi_partial 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_test_multi 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_test_sametarget 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_backfill_test_simple 2024-09-10T20:03:13.527 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_backfill_grow 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_backfill_multi 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:TEST_ec_backfill_simple' 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:36: run: for func in $funcs 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:37: run: setup td/osd-backfill-space 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/osd-backfill-space 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/osd-backfill-space 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-space 2024-09-10T20:03:13.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2024-09-10T20:03:13.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-space KILL 2024-09-10T20:03:13.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:03:13.529 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:03:13.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:03:13.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:03:13.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:03:13.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:03:13.530 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:03:13.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:03:13.532 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:03:13.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-space 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:13.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.115082 2024-09-10T20:03:13.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:03:13.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:03:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/osd-backfill-space 2024-09-10T20:03:13.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2024-09-10T20:03:13.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:13.535 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:13.535 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.115082 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/osd-backfill-space 1' TERM HUP INT 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:38: run: TEST_backfill_grow td/osd-backfill-space 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:558: TEST_backfill_grow: local dir=td/osd-backfill-space 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:559: TEST_backfill_grow: local poolname=test 2024-09-10T20:03:13.537 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:560: TEST_backfill_grow: local OSDS=3 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:562: TEST_backfill_grow: run_mon td/osd-backfill-space a 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/osd-backfill-space 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/osd-backfill-space/a 2024-09-10T20:03:13.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-space/a --run-dir=td/osd-backfill-space 2024-09-10T20:03:13.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2024-09-10T20:03:13.998 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:13.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-space/a '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-space/log --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-10T20:03:14.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2024-09-10T20:03:14.225 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.115082/ceph-mon.a.asok 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:03:14.226 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.115082/ceph-mon.a.asok config get fsid 2024-09-10T20:03:14.303 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.115082/ceph-mon.a.asok 2024-09-10T20:03:14.304 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2024-09-10T20:03:14.305 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.115082/ceph-mon.a.asok config get mon_host 2024-09-10T20:03:14.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:563: TEST_backfill_grow: run_mgr td/osd-backfill-space x 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/osd-backfill-space 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/osd-backfill-space/x 2024-09-10T20:03:14.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:15.238 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:03:15.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-space/x '--log-file=td/osd-backfill-space/$name.log' '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --run-dir=td/osd-backfill-space '--pid-file=td/osd-backfill-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-10T20:03:15.271 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:565: TEST_backfill_grow: expr 3 - 1 2024-09-10T20:03:15.272 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:565: TEST_backfill_grow: seq 0 2 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:565: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:567: TEST_backfill_grow: run_osd td/osd-backfill-space 0 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-space 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2024-09-10T20:03:15.276 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-space/0 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:15.277 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:03:15.278 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-space/0 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1' 2024-09-10T20:03:15.279 INFO:tasks.workunit.client.0.smithi130.stdout:add osd0 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 2024-09-10T20:03:15.280 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:03:15.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQADpuBm1qyzERAA9p/CvgseawfiDwO9TX1nLg== 2024-09-10T20:03:15.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQADpuBm1qyzERAA9p/CvgseawfiDwO9TX1nLg=="}' 2024-09-10T20:03:15.300 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 -i td/osd-backfill-space/0/new.json 2024-09-10T20:03:15.566 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:03:15.576 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-space/0/new.json 2024-09-10T20:03:15.577 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQADpuBm1qyzERAA9p/CvgseawfiDwO9TX1nLg== --osd-uuid 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 2024-09-10T20:03:15.613 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:15.610+0000 7fe89f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.634 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:15.631+0000 7fe89f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.635 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:15.632+0000 7fe89f614640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:15.636 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:15.633+0000 7fe89f614640 -1 bdev(0x55d921107c00 td/osd-backfill-space/0/block) open stat got: (1) Operation not permitted 2024-09-10T20:03:15.636 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:15.633+0000 7fe89f614640 -1 bluestore(td/osd-backfill-space/0) _read_fsid unparsable uuid 2024-09-10T20:03:21.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-space/0/keyring 2024-09-10T20:03:21.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:03:21.104 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd0 key to auth repository 2024-09-10T20:03:21.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2024-09-10T20:03:21.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:03:21.651 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T20:03:21.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2024-09-10T20:03:21.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:03:21.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:03:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:03:21.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:03:21.688 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:21.685+0000 7f0e1180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:21.690 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:21.687+0000 7f0e1180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:21.691 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:21.689+0000 7f0e1180d640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:22.085 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:03:22.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:22.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:22.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:22.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:23.282 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:23.280+0000 7f0e1180d640 -1 Falling back to public interface 2024-09-10T20:03:23.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:23.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:23.475 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:03:23.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:03:23.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:23.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:23.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:24.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:24.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:24.876 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:03:24.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:03:24.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:24.877 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:25.098 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:25.096+0000 7f0e1180d640 -1 osd.0 0 log_to_monitors true 2024-09-10T20:03:25.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:26.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:26.367 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:26.367 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:03:26.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:03:26.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:26.368 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:26.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:27.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:27.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:27.873 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:03:27.874 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:03:27.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:27.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:28.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:29.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:29.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:29.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:03:29.315 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:03:29.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:29.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:29.867 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:30.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:30.868 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:30.869 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:03:30.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:03:30.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:30.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:31.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:32.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:32.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:32.299 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:03:32.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:03:32.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:32.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:32.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:33.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:33.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:33.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:03:33.880 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:03:33.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:33.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:34.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:35.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:35.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:35.309 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:03:35.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:03:35.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:35.309 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:35.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:36.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:36.885 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:36.885 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:03:36.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:03:36.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:36.886 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:37.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:38.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:38.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:38.821 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:03:38.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:03:38.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:38.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:39.395 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:40.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:40.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:40.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:03:40.397 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:03:40.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:40.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:41.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:42.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:42.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:42.068 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:03:42.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:03:42.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:42.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:42.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:43.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:43.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:43.896 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:03:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:03:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:43.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:44.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:45.892 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:45.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:45.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:03:45.893 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:03:45.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:45.893 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:46.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:47.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:47.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:47.945 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:03:47.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:03:47.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:47.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:03:48.399 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3051533156,v1:127.0.0.1:6803/3051533156] [v2:127.0.0.1:6804/3051533156,v1:127.0.0.1:6805/3051533156] exists,up 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:565: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:567: TEST_backfill_grow: run_osd td/osd-backfill-space 1 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-space 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:03:48.403 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-space/1 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/1' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/1/journal' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2024-09-10T20:03:48.404 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2024-09-10T20:03:48.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-space/1 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:03:48.406 INFO:tasks.workunit.client.0.smithi130.stdout:add osd1 38548c14-ce4c-4e4f-860f-584afd7fd3b3 2024-09-10T20:03:48.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=38548c14-ce4c-4e4f-860f-584afd7fd3b3 2024-09-10T20:03:48.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 38548c14-ce4c-4e4f-860f-584afd7fd3b3' 2024-09-10T20:03:48.407 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:03:48.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAkpuBmm1g9GRAAcf3oIT61Vh1WjmKZcEdFEg== 2024-09-10T20:03:48.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAkpuBmm1g9GRAAcf3oIT61Vh1WjmKZcEdFEg=="}' 2024-09-10T20:03:48.426 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 38548c14-ce4c-4e4f-860f-584afd7fd3b3 -i td/osd-backfill-space/1/new.json 2024-09-10T20:03:48.897 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:03:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-space/1/new.json 2024-09-10T20:03:48.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkpuBmm1g9GRAAcf3oIT61Vh1WjmKZcEdFEg== --osd-uuid 38548c14-ce4c-4e4f-860f-584afd7fd3b3 2024-09-10T20:03:48.943 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:48.941+0000 7f003bd34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:48.968 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:48.964+0000 7f003bd34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:48.969 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:48.965+0000 7f003bd34640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:48.969 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:48.965+0000 7f003bd34640 -1 bdev(0x558f97773c00 td/osd-backfill-space/1/block) open stat got: (1) Operation not permitted 2024-09-10T20:03:48.969 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:48.965+0000 7f003bd34640 -1 bluestore(td/osd-backfill-space/1) _read_fsid unparsable uuid 2024-09-10T20:03:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-space/1/keyring 2024-09-10T20:03:54.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:03:54.748 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd1 key to auth repository 2024-09-10T20:03:54.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2024-09-10T20:03:54.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:03:55.289 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.1 2024-09-10T20:03:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2024-09-10T20:03:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/1 --osd-journal=td/osd-backfill-space/1/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:03:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:03:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:03:55.290 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:03:55.324 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:55.322+0000 7f213400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:55.336 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:55.334+0000 7f213400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:55.338 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:55.335+0000 7f213400b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:03:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2024-09-10T20:03:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:03:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2024-09-10T20:03:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:03:55.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:03:55.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:55.709 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:03:55.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:03:55.709 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:55.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:56.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:56.680 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:56.678+0000 7f213400b640 -1 Falling back to public interface 2024-09-10T20:03:57.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:57.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:57.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:03:57.096 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:03:57.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:57.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:57.529 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:03:58.280 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:03:58.278+0000 7f213400b640 -1 osd.1 0 log_to_monitors true 2024-09-10T20:03:58.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:03:58.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:03:58.531 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:03:58.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:03:58.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:03:58.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:03:59.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:00.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:00.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:00.115 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:04:00.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:04:00.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:00.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:00.538 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:01.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:01.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:01.540 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:04:01.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:04:01.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:01.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:02.139 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:03.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:03.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:03.141 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:04:03.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:04:03.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:03.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:03.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:04.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:04.675 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:04.675 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:04:04.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:04:04.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:04.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:05.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:06.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:06.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:06.236 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:04:06.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:04:06.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:06.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:06.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:07.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:07.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:07.659 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:04:07.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:04:07.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:07.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:08.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:09.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:09.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:09.101 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:04:09.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:04:09.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:09.101 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:09.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:10.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:10.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:10.632 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:04:10.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:04:10.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:10.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:11.519 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:12.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:12.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:12.521 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:04:12.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:04:12.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:12.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:13.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:14.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:14.038 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:04:14.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:14.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:04:14.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:14.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:14.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:15.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:15.797 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:04:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:04:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:15.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:16.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:17.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:17.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:17.994 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:04:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:04:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:17.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:18.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:19.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:19.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:19.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:04:19.947 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:04:19.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:19.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:20.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:21.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:21.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:21.860 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:04:21.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:04:21.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:21.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2024-09-10T20:04:22.249 INFO:tasks.workunit.client.0.smithi130.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1629639663,v1:127.0.0.1:6811/1629639663] [v2:127.0.0.1:6812/1629639663,v1:127.0.0.1:6813/1629639663] exists,up 38548c14-ce4c-4e4f-860f-584afd7fd3b3 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:565: TEST_backfill_grow: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:567: TEST_backfill_grow: run_osd td/osd-backfill-space 2 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/osd-backfill-space 2024-09-10T20:04:22.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/osd-backfill-space/2 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/osd-backfill-space/2' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/2/journal' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2024-09-10T20:04:22.254 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2024-09-10T20:04:22.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/osd-backfill-space/2 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0af4aaa6-f1ce-46ef-bd8c-ffababc00e71 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 0af4aaa6-f1ce-46ef-bd8c-ffababc00e71' 2024-09-10T20:04:22.256 INFO:tasks.workunit.client.0.smithi130.stdout:add osd2 0af4aaa6-f1ce-46ef-bd8c-ffababc00e71 2024-09-10T20:04:22.257 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2024-09-10T20:04:22.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBGpuBma8haEBAAKa90vxSFrknUPsH3xoaFeg== 2024-09-10T20:04:22.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBGpuBma8haEBAAKa90vxSFrknUPsH3xoaFeg=="}' 2024-09-10T20:04:22.277 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0af4aaa6-f1ce-46ef-bd8c-ffababc00e71 -i td/osd-backfill-space/2/new.json 2024-09-10T20:04:22.769 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:04:22.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/osd-backfill-space/2/new.json 2024-09-10T20:04:22.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGpuBma8haEBAAKa90vxSFrknUPsH3xoaFeg== --osd-uuid 0af4aaa6-f1ce-46ef-bd8c-ffababc00e71 2024-09-10T20:04:22.817 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:22.814+0000 7f5a9d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:22.838 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:22.835+0000 7f5a9d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:22.836+0000 7f5a9d20b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:22.839 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:22.837+0000 7f5a9d20b640 -1 bdev(0x557d87b0dc00 td/osd-backfill-space/2/block) open stat got: (1) Operation not permitted 2024-09-10T20:04:22.840 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:22.837+0000 7f5a9d20b640 -1 bluestore(td/osd-backfill-space/2) _read_fsid unparsable uuid 2024-09-10T20:04:28.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/osd-backfill-space/2/keyring 2024-09-10T20:04:28.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2024-09-10T20:04:28.828 INFO:tasks.workunit.client.0.smithi130.stdout:adding osd2 key to auth repository 2024-09-10T20:04:28.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2024-09-10T20:04:28.829 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/osd-backfill-space/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-10T20:04:29.368 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.2 2024-09-10T20:04:29.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2024-09-10T20:04:29.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/2 --osd-journal=td/osd-backfill-space/2/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:04:29.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2024-09-10T20:04:29.369 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2024-09-10T20:04:29.370 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2024-09-10T20:04:29.403 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:29.401+0000 7f2cb6416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:29.408 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:29.405+0000 7f2cb6416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:29.409 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:29.407+0000 7f2cb6416640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:04:29.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2024-09-10T20:04:29.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:04:29.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:04:29.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:29.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:30.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:30.754 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:30.752+0000 7f2cb6416640 -1 Falling back to public interface 2024-09-10T20:04:31.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:31.249 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:04:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:04:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:31.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:31.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:32.353 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:04:32.351+0000 7f2cb6416640 -1 osd.2 0 log_to_monitors true 2024-09-10T20:04:32.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:32.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:32.694 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:04:32.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:04:32.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:32.694 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:33.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:34.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:34.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:34.166 INFO:tasks.workunit.client.0.smithi130.stdout:3 2024-09-10T20:04:34.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2024-09-10T20:04:34.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:34.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:34.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:35.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:35.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:35.582 INFO:tasks.workunit.client.0.smithi130.stdout:4 2024-09-10T20:04:35.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 4 2024-09-10T20:04:35.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:35.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:36.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:37.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:37.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:37.057 INFO:tasks.workunit.client.0.smithi130.stdout:5 2024-09-10T20:04:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 5 2024-09-10T20:04:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:37.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:37.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:38.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:38.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:38.462 INFO:tasks.workunit.client.0.smithi130.stdout:6 2024-09-10T20:04:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 6 2024-09-10T20:04:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:38.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:38.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:39.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:39.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:39.861 INFO:tasks.workunit.client.0.smithi130.stdout:7 2024-09-10T20:04:39.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 7 2024-09-10T20:04:39.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:39.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:40.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:41.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:41.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:41.331 INFO:tasks.workunit.client.0.smithi130.stdout:8 2024-09-10T20:04:41.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 8 2024-09-10T20:04:41.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:41.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:41.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:42.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:42.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:42.841 INFO:tasks.workunit.client.0.smithi130.stdout:9 2024-09-10T20:04:42.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 9 2024-09-10T20:04:42.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:42.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:43.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:44.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 10 2024-09-10T20:04:44.352 INFO:tasks.workunit.client.0.smithi130.stdout:10 2024-09-10T20:04:44.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:44.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:44.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:45.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:45.750 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:45.751 INFO:tasks.workunit.client.0.smithi130.stdout:11 2024-09-10T20:04:45.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 11 2024-09-10T20:04:45.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:45.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:46.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:47.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:47.679 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:47.680 INFO:tasks.workunit.client.0.smithi130.stdout:12 2024-09-10T20:04:47.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 12 2024-09-10T20:04:47.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:47.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:48.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:49.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:49.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:49.320 INFO:tasks.workunit.client.0.smithi130.stdout:13 2024-09-10T20:04:49.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 13 2024-09-10T20:04:49.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:49.320 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:49.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:50.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:50.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:50.802 INFO:tasks.workunit.client.0.smithi130.stdout:14 2024-09-10T20:04:50.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 14 2024-09-10T20:04:50.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:50.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:51.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:52.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:52.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:52.486 INFO:tasks.workunit.client.0.smithi130.stdout:15 2024-09-10T20:04:52.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 15 2024-09-10T20:04:52.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:52.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:53.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:54.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:54.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:54.230 INFO:tasks.workunit.client.0.smithi130.stdout:16 2024-09-10T20:04:54.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 16 2024-09-10T20:04:54.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:54.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:55.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:04:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:04:56.098 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:04:56.099 INFO:tasks.workunit.client.0.smithi130.stdout:17 2024-09-10T20:04:56.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 17 2024-09-10T20:04:56.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:04:56.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2024-09-10T20:04:56.543 INFO:tasks.workunit.client.0.smithi130.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2083059260,v1:127.0.0.1:6819/2083059260] [v2:127.0.0.1:6820/2083059260,v1:127.0.0.1:6821/2083059260] exists,up 0af4aaa6-f1ce-46ef-bd8c-ffababc00e71 2024-09-10T20:04:56.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:04:56.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:04:56.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:04:56.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:570: TEST_backfill_grow: ceph osd set-backfillfull-ratio .85 2024-09-10T20:04:57.177 INFO:tasks.workunit.client.0.smithi130.stderr:osd set-backfillfull-ratio 0.85 2024-09-10T20:04:57.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:572: TEST_backfill_grow: create_pool test 1 1 2024-09-10T20:04:57.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 1 1 2024-09-10T20:04:57.730 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' already exists 2024-09-10T20:04:57.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2024-09-10T20:04:58.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:573: TEST_backfill_grow: ceph osd pool set test size 3 2024-09-10T20:04:59.655 INFO:tasks.workunit.client.0.smithi130.stderr:set pool 1 size to 3 2024-09-10T20:04:59.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:574: TEST_backfill_grow: sleep 5 2024-09-10T20:05:04.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:576: TEST_backfill_grow: wait_for_clean 2024-09-10T20:05:04.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:05:04.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:05:04.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:05:04.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:05:04.672 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:05:04.672 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:05:04.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:05:04.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:05:04.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:05:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:05:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:05:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:05:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:05:04.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:05:04.767 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:05.200 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:05:05.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=21474836501 2024-09-10T20:05:05.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 21474836501 2024-09-10T20:05:05.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501' 2024-09-10T20:05:05.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:05.319 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:05:05.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949672976 2024-09-10T20:05:05.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949672976 2024-09-10T20:05:05.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672976' 2024-09-10T20:05:05.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:05:05.440 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:05:05.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509448 2024-09-10T20:05:05.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509448 2024-09-10T20:05:05.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-21474836501 1-42949672976 2-64424509448' 2024-09-10T20:05:05.593 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:05.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-21474836501 2024-09-10T20:05:05.594 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:05.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:05:05.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-21474836501 2024-09-10T20:05:05.595 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:05.596 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 21474836501 2024-09-10T20:05:05.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=21474836501 2024-09-10T20:05:05.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-10T20:05:05.597 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:06.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-10T20:05:06.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:05:07.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:05:07.028 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:05:07.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-10T20:05:07.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:07.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949672976 2024-09-10T20:05:07.445 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:07.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:05:07.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949672976 2024-09-10T20:05:07.446 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:07.447 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949672976 2024-09-10T20:05:07.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949672976 2024-09-10T20:05:07.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-10T20:05:07.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:05:07.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-10T20:05:07.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:05:07.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509448 2024-09-10T20:05:07.880 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:05:07.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:05:07.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509448 2024-09-10T20:05:07.881 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:05:07.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509448 2024-09-10T20:05:07.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-10T20:05:07.883 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509448 2024-09-10T20:05:07.883 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:05:08.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509449 -lt 64424509448 2024-09-10T20:05:08.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:05:08.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:08.274 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:05:08.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:05:08.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:05:08.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:05:08.801 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:05:08.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:05:08.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:05:08.802 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:05:09.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:05:09.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:05:09.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:05:09.219 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:05:09.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:05:09.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:578: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/4kdata bs=1k count=4 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records in 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr:4+0 records out 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 6.0667e-05 s, 67.5 MB/s 2024-09-10T20:05:09.741 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: seq 1 600 2024-09-10T20:05:09.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:09.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj1 td/osd-backfill-space/4kdata 2024-09-10T20:05:09.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:09.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj2 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.042 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj3 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.184 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj4 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj5 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.452 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj6 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj7 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj8 td/osd-backfill-space/4kdata 2024-09-10T20:05:10.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:10.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj9 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.030 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.031 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj10 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj11 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.326 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj12 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.500 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj13 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj14 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj15 td/osd-backfill-space/4kdata 2024-09-10T20:05:11.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:11.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj16 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj17 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj18 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj19 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj20 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj21 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj22 td/osd-backfill-space/4kdata 2024-09-10T20:05:12.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:12.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj23 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.084 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj24 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj25 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj26 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj27 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj28 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj29 td/osd-backfill-space/4kdata 2024-09-10T20:05:13.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:13.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj30 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj31 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.283 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj32 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj33 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj34 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.710 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj35 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj36 td/osd-backfill-space/4kdata 2024-09-10T20:05:14.994 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:14.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj37 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj38 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj39 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj40 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj41 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj42 td/osd-backfill-space/4kdata 2024-09-10T20:05:15.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:15.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj43 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj44 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.218 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj45 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj46 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj47 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj48 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj49 td/osd-backfill-space/4kdata 2024-09-10T20:05:16.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:16.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj50 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj51 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.267 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj52 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.434 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj53 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj54 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.769 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj55 td/osd-backfill-space/4kdata 2024-09-10T20:05:17.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:17.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj56 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.053 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj57 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj58 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj59 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.481 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj60 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.623 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.624 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj61 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj62 td/osd-backfill-space/4kdata 2024-09-10T20:05:18.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:18.909 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj63 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.099 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj64 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj65 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.511 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj66 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj67 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj68 td/osd-backfill-space/4kdata 2024-09-10T20:05:19.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:19.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj69 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj70 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj71 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.382 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj72 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj73 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj74 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.810 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj75 td/osd-backfill-space/4kdata 2024-09-10T20:05:20.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:20.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj76 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.094 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj77 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.237 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj78 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj79 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj80 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.724 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj81 td/osd-backfill-space/4kdata 2024-09-10T20:05:21.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:21.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj82 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj83 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.176 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.177 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj84 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj85 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj86 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.604 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj87 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj88 td/osd-backfill-space/4kdata 2024-09-10T20:05:22.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:22.897 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj89 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj90 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj91 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj92 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.525 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj93 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.676 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj94 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj95 td/osd-backfill-space/4kdata 2024-09-10T20:05:23.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:23.961 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj96 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj97 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.304 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj98 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj99 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj100 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj101 td/osd-backfill-space/4kdata 2024-09-10T20:05:24.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:24.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj102 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj103 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj104 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.335 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj105 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj106 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj107 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.805 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj108 td/osd-backfill-space/4kdata 2024-09-10T20:05:25.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:25.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj109 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj110 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj111 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj112 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj113 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj114 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj115 td/osd-backfill-space/4kdata 2024-09-10T20:05:26.943 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:26.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj116 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj117 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.228 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj118 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.421 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj119 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj120 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj121 td/osd-backfill-space/4kdata 2024-09-10T20:05:27.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:27.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj122 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.058 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj123 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj124 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj125 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj126 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj127 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj128 td/osd-backfill-space/4kdata 2024-09-10T20:05:28.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:28.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj129 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj130 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj131 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj132 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj133 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj134 td/osd-backfill-space/4kdata 2024-09-10T20:05:29.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:29.918 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj135 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.061 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj136 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj137 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.364 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.365 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj138 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj139 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj140 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj141 td/osd-backfill-space/4kdata 2024-09-10T20:05:30.923 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:30.924 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj142 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj143 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj144 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj145 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.503 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj146 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj147 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.787 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj148 td/osd-backfill-space/4kdata 2024-09-10T20:05:31.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:31.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj149 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj150 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.809 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/4kdata 2024-09-10T20:05:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:32.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/4kdata 2024-09-10T20:05:33.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:33.874 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/4kdata 2024-09-10T20:05:34.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:34.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.659 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.801 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/4kdata 2024-09-10T20:05:35.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:35.944 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.086 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.547 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/4kdata 2024-09-10T20:05:36.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:36.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.142 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.301 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.636 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.795 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/4kdata 2024-09-10T20:05:37.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:37.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.089 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.374 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.667 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.818 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/4kdata 2024-09-10T20:05:38.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:38.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.381 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/4kdata 2024-09-10T20:05:39.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:39.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.318 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.460 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.602 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/4kdata 2024-09-10T20:05:40.895 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:40.896 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.047 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.507 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.651 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.793 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/4kdata 2024-09-10T20:05:41.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:41.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.077 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/4kdata 2024-09-10T20:05:42.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:42.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.410 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.737 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/4kdata 2024-09-10T20:05:43.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:43.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.776 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/4kdata 2024-09-10T20:05:44.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:44.927 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.665 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/4kdata 2024-09-10T20:05:45.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:45.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.536 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/4kdata 2024-09-10T20:05:46.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:46.946 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.116 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.409 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.610 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.770 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/4kdata 2024-09-10T20:05:47.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:47.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.054 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.490 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/4kdata 2024-09-10T20:05:48.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:48.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/4kdata 2024-09-10T20:05:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:49.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/4kdata 2024-09-10T20:05:49.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:49.219 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/4kdata 2024-09-10T20:05:49.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:49.378 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/4kdata 2024-09-10T20:05:49.554 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:49.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/4kdata 2024-09-10T20:05:49.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:49.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/4kdata 2024-09-10T20:05:50.215 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:50.216 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/4kdata 2024-09-10T20:05:50.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:50.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/4kdata 2024-09-10T20:05:50.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:50.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/4kdata 2024-09-10T20:05:50.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:50.701 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/4kdata 2024-09-10T20:05:50.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:50.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.027 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.169 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.311 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.462 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/4kdata 2024-09-10T20:05:51.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:51.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.376 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.668 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/4kdata 2024-09-10T20:05:52.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:52.962 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.272 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.607 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.775 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/4kdata 2024-09-10T20:05:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/4kdata 2024-09-10T20:05:54.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:54.931 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/4kdata 2024-09-10T20:05:55.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:55.878 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.162 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/4kdata 2024-09-10T20:05:56.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:56.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.059 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.201 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.486 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.804 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/4kdata 2024-09-10T20:05:57.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:57.980 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.407 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.550 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.692 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.693 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/4kdata 2024-09-10T20:05:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:58.977 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.119 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.270 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/4kdata 2024-09-10T20:05:59.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:05:59.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.360 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/4kdata 2024-09-10T20:06:00.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:00.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/4kdata 2024-09-10T20:06:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:01.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.203 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.362 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/4kdata 2024-09-10T20:06:02.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:02.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.400 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.861 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/4kdata 2024-09-10T20:06:03.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:03.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.280 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.565 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/4kdata 2024-09-10T20:06:04.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:04.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.001 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.294 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.630 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/4kdata 2024-09-10T20:06:05.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:05.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.484 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.660 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.802 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.803 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/4kdata 2024-09-10T20:06:06.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:06.945 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.095 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.096 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.397 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.733 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/4kdata 2024-09-10T20:06:07.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:07.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.620 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/4kdata 2024-09-10T20:06:08.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:08.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.609 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.751 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/4kdata 2024-09-10T20:06:09.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:09.894 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.036 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.037 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.179 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.338 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/4kdata 2024-09-10T20:06:10.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:10.933 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.234 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.570 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/4kdata 2024-09-10T20:06:11.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:11.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.038 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/4kdata 2024-09-10T20:06:12.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:12.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.044 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.513 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.664 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/4kdata 2024-09-10T20:06:13.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:13.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.543 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.544 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/4kdata 2024-09-10T20:06:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:14.970 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/4kdata 2024-09-10T20:06:15.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:15.884 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.043 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.186 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.328 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.797 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/4kdata 2024-09-10T20:06:16.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:16.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.585 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.727 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/4kdata 2024-09-10T20:06:17.869 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:17.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.154 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.313 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.314 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.790 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/4kdata 2024-09-10T20:06:18.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:18.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.402 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.729 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/4kdata 2024-09-10T20:06:19.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:19.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.181 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.324 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.466 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.626 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.768 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/4kdata 2024-09-10T20:06:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:20.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/4kdata 2024-09-10T20:06:21.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:21.418 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/4kdata 2024-09-10T20:06:21.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:21.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/4kdata 2024-09-10T20:06:21.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:21.757 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/4kdata 2024-09-10T20:06:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:21.915 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.200 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.492 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.493 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.635 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.777 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/4kdata 2024-09-10T20:06:22.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:22.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.078 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.079 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.422 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/4kdata 2024-09-10T20:06:23.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:23.925 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.210 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.494 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.645 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/4kdata 2024-09-10T20:06:24.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:24.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.684 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.827 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/4kdata 2024-09-10T20:06:25.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:25.969 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.111 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.413 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/4kdata 2024-09-10T20:06:26.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:26.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.401 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.720 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/4kdata 2024-09-10T20:06:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:27.870 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.616 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.784 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/4kdata 2024-09-10T20:06:28.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:28.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.068 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.211 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.353 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.496 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.655 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/4kdata 2024-09-10T20:06:29.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:29.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.090 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.526 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.677 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.819 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.820 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/4kdata 2024-09-10T20:06:30.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:30.954 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/4kdata 2024-09-10T20:06:31.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:31.917 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.202 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.345 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.495 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.654 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/4kdata 2024-09-10T20:06:32.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:32.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.315 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.316 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.475 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.785 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/4kdata 2024-09-10T20:06:33.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:33.936 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.069 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.220 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.815 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/4kdata 2024-09-10T20:06:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:34.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.108 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.586 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/4kdata 2024-09-10T20:06:35.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:35.871 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.448 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.742 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/4kdata 2024-09-10T20:06:36.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:36.876 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.009 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.010 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.152 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.319 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.629 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.879 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/4kdata 2024-09-10T20:06:37.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:37.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.097 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.206 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.323 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/4kdata 2024-09-10T20:06:38.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:38.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.310 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.594 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.595 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.703 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.812 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/4kdata 2024-09-10T20:06:39.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:39.921 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/4kdata 2024-09-10T20:06:40.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:579: TEST_backfill_grow: for i in $(seq 1 $objects) 2024-09-10T20:06:40.046 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:581: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/4kdata 2024-09-10T20:06:40.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:584: TEST_backfill_grow: get_pg test obj1 2024-09-10T20:06:40.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=test 2024-09-10T20:06:40.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=obj1 2024-09-10T20:06:40.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map test obj1 2024-09-10T20:06:40.156 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2024-09-10T20:06:40.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:584: TEST_backfill_grow: local PG=1.0 2024-09-10T20:06:40.584 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:586: TEST_backfill_grow: get_primary test obj1 2024-09-10T20:06:40.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T20:06:40.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T20:06:40.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T20:06:40.585 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:06:41.018 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:586: TEST_backfill_grow: local primary=1 2024-09-10T20:06:41.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:587: TEST_backfill_grow: get_not_primary test obj1 2024-09-10T20:06:41.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=test 2024-09-10T20:06:41.019 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=obj1 2024-09-10T20:06:41.019 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary test obj1 2024-09-10T20:06:41.019 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=test 2024-09-10T20:06:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=obj1 2024-09-10T20:06:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map test obj1 2024-09-10T20:06:41.020 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2024-09-10T20:06:41.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2024-09-10T20:06:41.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map test obj1 2024-09-10T20:06:41.419 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-10T20:06:41.822 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:587: TEST_backfill_grow: local otherosd=0 2024-09-10T20:06:41.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:589: TEST_backfill_grow: ceph osd set noout 2024-09-10T20:06:42.461 INFO:tasks.workunit.client.0.smithi130.stderr:noout is set 2024-09-10T20:06:42.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:590: TEST_backfill_grow: kill_daemons td/osd-backfill-space TERM 0 2024-09-10T20:06:42.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:06:42.473 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:06:42.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:06:42.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:06:42.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:06:42.780 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:06:42.780 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:592: TEST_backfill_grow: expr 600 / 4 2024-09-10T20:06:42.781 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:592: TEST_backfill_grow: rmobjects=150 2024-09-10T20:06:42.781 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: seq 1 150 2024-09-10T20:06:42.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:42.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj1 2024-09-10T20:06:43.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.352 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj2 2024-09-10T20:06:43.468 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.469 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj3 2024-09-10T20:06:43.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj4 2024-09-10T20:06:43.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj5 2024-09-10T20:06:43.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj6 2024-09-10T20:06:43.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj7 2024-09-10T20:06:43.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:43.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj8 2024-09-10T20:06:44.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.104 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj9 2024-09-10T20:06:44.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj10 2024-09-10T20:06:44.287 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.288 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj11 2024-09-10T20:06:44.379 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj12 2024-09-10T20:06:44.471 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj13 2024-09-10T20:06:44.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.563 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj14 2024-09-10T20:06:44.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj15 2024-09-10T20:06:44.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj16 2024-09-10T20:06:44.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj17 2024-09-10T20:06:44.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:44.973 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj18 2024-09-10T20:06:45.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj19 2024-09-10T20:06:45.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj20 2024-09-10T20:06:45.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.266 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj21 2024-09-10T20:06:45.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj22 2024-09-10T20:06:45.578 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj23 2024-09-10T20:06:45.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.617 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj24 2024-09-10T20:06:45.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.759 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj25 2024-09-10T20:06:45.859 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.860 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj26 2024-09-10T20:06:45.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:45.960 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj27 2024-09-10T20:06:46.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj28 2024-09-10T20:06:46.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.160 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj29 2024-09-10T20:06:46.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.260 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj30 2024-09-10T20:06:46.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.361 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj31 2024-09-10T20:06:46.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj32 2024-09-10T20:06:46.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.561 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj33 2024-09-10T20:06:46.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.653 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj34 2024-09-10T20:06:46.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.745 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj35 2024-09-10T20:06:46.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.837 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj36 2024-09-10T20:06:46.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:46.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj37 2024-09-10T20:06:47.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj38 2024-09-10T20:06:47.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.113 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj39 2024-09-10T20:06:47.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj40 2024-09-10T20:06:47.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj41 2024-09-10T20:06:47.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj42 2024-09-10T20:06:47.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj43 2024-09-10T20:06:47.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj44 2024-09-10T20:06:47.740 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj45 2024-09-10T20:06:47.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj46 2024-09-10T20:06:47.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:47.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj47 2024-09-10T20:06:48.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.041 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj48 2024-09-10T20:06:48.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj49 2024-09-10T20:06:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.242 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj50 2024-09-10T20:06:48.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.351 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj51 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.451 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj52 2024-09-10T20:06:48.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.551 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj53 2024-09-10T20:06:48.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.652 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj54 2024-09-10T20:06:48.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.752 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj55 2024-09-10T20:06:48.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.852 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj56 2024-09-10T20:06:48.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:48.952 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj57 2024-09-10T20:06:49.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.052 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj58 2024-09-10T20:06:49.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.153 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj59 2024-09-10T20:06:49.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.253 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj60 2024-09-10T20:06:49.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj61 2024-09-10T20:06:49.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.488 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj62 2024-09-10T20:06:49.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj63 2024-09-10T20:06:49.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj64 2024-09-10T20:06:49.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj65 2024-09-10T20:06:49.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj66 2024-09-10T20:06:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:49.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj67 2024-09-10T20:06:50.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj68 2024-09-10T20:06:50.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.148 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj69 2024-09-10T20:06:50.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj70 2024-09-10T20:06:50.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj71 2024-09-10T20:06:50.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj72 2024-09-10T20:06:50.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj73 2024-09-10T20:06:50.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.641 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj74 2024-09-10T20:06:50.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj75 2024-09-10T20:06:50.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.842 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj76 2024-09-10T20:06:50.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:50.942 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj77 2024-09-10T20:06:51.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.067 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj78 2024-09-10T20:06:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj79 2024-09-10T20:06:51.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj80 2024-09-10T20:06:51.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.393 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj81 2024-09-10T20:06:51.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj82 2024-09-10T20:06:51.627 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj83 2024-09-10T20:06:51.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj84 2024-09-10T20:06:51.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj85 2024-09-10T20:06:51.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:51.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj86 2024-09-10T20:06:52.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.029 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj87 2024-09-10T20:06:52.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.121 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj88 2024-09-10T20:06:52.212 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.213 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj89 2024-09-10T20:06:52.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj90 2024-09-10T20:06:52.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj91 2024-09-10T20:06:52.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj92 2024-09-10T20:06:52.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj93 2024-09-10T20:06:52.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:52.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj94 2024-09-10T20:06:53.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj95 2024-09-10T20:06:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj96 2024-09-10T20:06:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.279 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj97 2024-09-10T20:06:53.398 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj98 2024-09-10T20:06:53.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj99 2024-09-10T20:06:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj100 2024-09-10T20:06:53.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.725 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj101 2024-09-10T20:06:53.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj102 2024-09-10T20:06:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:53.926 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj103 2024-09-10T20:06:54.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj104 2024-09-10T20:06:54.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.126 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj105 2024-09-10T20:06:54.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.227 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj106 2024-09-10T20:06:54.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.327 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj107 2024-09-10T20:06:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.427 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj108 2024-09-10T20:06:54.527 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.528 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj109 2024-09-10T20:06:54.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.628 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj110 2024-09-10T20:06:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.728 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj111 2024-09-10T20:06:54.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.828 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj112 2024-09-10T20:06:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:54.928 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj113 2024-09-10T20:06:55.020 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj114 2024-09-10T20:06:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.112 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj115 2024-09-10T20:06:55.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj116 2024-09-10T20:06:55.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.305 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj117 2024-09-10T20:06:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.405 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj118 2024-09-10T20:06:55.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj119 2024-09-10T20:06:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj120 2024-09-10T20:06:55.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.731 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj121 2024-09-10T20:06:55.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj122 2024-09-10T20:06:55.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:55.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj123 2024-09-10T20:06:56.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.032 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj124 2024-09-10T20:06:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj125 2024-09-10T20:06:56.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.233 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj126 2024-09-10T20:06:56.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj127 2024-09-10T20:06:56.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.433 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj128 2024-09-10T20:06:56.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.542 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj129 2024-09-10T20:06:56.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.642 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj130 2024-09-10T20:06:56.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.743 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj131 2024-09-10T20:06:56.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.835 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj132 2024-09-10T20:06:56.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:56.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj133 2024-09-10T20:06:57.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj134 2024-09-10T20:06:57.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj135 2024-09-10T20:06:57.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.235 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj136 2024-09-10T20:06:57.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.386 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj137 2024-09-10T20:06:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.487 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj138 2024-09-10T20:06:57.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.579 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj139 2024-09-10T20:06:57.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj140 2024-09-10T20:06:57.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj141 2024-09-10T20:06:57.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.855 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj142 2024-09-10T20:06:57.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:57.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj143 2024-09-10T20:06:58.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj144 2024-09-10T20:06:58.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.147 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj145 2024-09-10T20:06:58.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj146 2024-09-10T20:06:58.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj147 2024-09-10T20:06:58.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj148 2024-09-10T20:06:58.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj149 2024-09-10T20:06:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:593: TEST_backfill_grow: for i in $(seq 1 $rmobjects) 2024-09-10T20:06:58.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:595: TEST_backfill_grow: rados -p test rm obj150 2024-09-10T20:06:58.741 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:598: TEST_backfill_grow: dd if=/dev/urandom of=td/osd-backfill-space/6kdata bs=6k count=1 2024-09-10T20:06:58.741 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records in 2024-09-10T20:06:58.741 INFO:tasks.workunit.client.0.smithi130.stderr:1+0 records out 2024-09-10T20:06:58.741 INFO:tasks.workunit.client.0.smithi130.stderr:6144 bytes (6.1 kB, 6.0 KiB) copied, 6.0728e-05 s, 101 MB/s 2024-09-10T20:06:58.742 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: expr 150 + 1 2024-09-10T20:06:58.743 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: seq 151 600 2024-09-10T20:06:58.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:58.744 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj151 td/osd-backfill-space/6kdata 2024-09-10T20:06:58.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:58.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj152 td/osd-backfill-space/6kdata 2024-09-10T20:06:58.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:58.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj153 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj154 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.151 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj155 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.251 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj156 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.384 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj157 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.510 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj158 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.619 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj159 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.711 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj160 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.811 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj161 td/osd-backfill-space/6kdata 2024-09-10T20:06:59.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:06:59.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj162 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj163 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj164 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.204 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj165 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj166 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.387 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj167 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.480 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj168 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.571 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj169 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj170 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj171 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj172 td/osd-backfill-space/6kdata 2024-09-10T20:07:00.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:00.981 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj173 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.081 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj174 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj175 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.291 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj176 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.408 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj177 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.534 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj178 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.634 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj179 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.734 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj180 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.834 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj181 td/osd-backfill-space/6kdata 2024-09-10T20:07:01.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:01.935 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj182 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.035 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj183 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.135 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj184 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj185 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.336 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj186 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj187 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.520 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj188 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.603 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj189 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.695 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj190 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.778 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj191 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj192 td/osd-backfill-space/6kdata 2024-09-10T20:07:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:02.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj193 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj194 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.129 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj195 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj196 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.269 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj197 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.346 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj198 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.446 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj199 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.521 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj200 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj201 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.663 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj202 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj203 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.796 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj204 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj205 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.929 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj206 td/osd-backfill-space/6kdata 2024-09-10T20:07:03.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:03.996 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj207 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj208 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.130 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj209 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.196 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj210 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj211 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.329 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj212 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj213 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.463 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj214 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj215 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj216 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj217 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj218 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj219 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.888 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj220 td/osd-backfill-space/6kdata 2024-09-10T20:07:04.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:04.955 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj221 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj222 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj223 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.155 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj224 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.221 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj225 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.296 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj226 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.363 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj227 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj228 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.530 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj229 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.596 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.597 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj230 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj231 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:05.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj232 td/osd-backfill-space/6kdata 2024-09-10T20:07:05.838 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj233 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj234 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj235 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj236 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.088 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj237 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.146 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj238 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.205 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj239 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj240 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj241 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.380 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj242 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.438 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj243 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.497 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj244 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.555 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj245 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.613 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj246 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj247 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.730 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj248 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.788 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj249 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.847 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj250 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj251 td/osd-backfill-space/6kdata 2024-09-10T20:07:06.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:06.963 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj252 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.022 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj253 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.080 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj254 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.138 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj255 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj256 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj257 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj258 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.455 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj259 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj260 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.580 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj261 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.638 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj262 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.697 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj263 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.755 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj264 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.813 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj265 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj266 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj267 td/osd-backfill-space/6kdata 2024-09-10T20:07:07.988 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:07.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj268 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.062 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj269 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj270 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.188 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.189 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj271 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.255 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj272 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj273 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.388 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.389 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj274 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj275 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.506 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj276 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.564 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj277 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.621 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj278 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.680 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj279 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.763 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj280 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.821 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj281 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.880 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj282 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.938 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj283 td/osd-backfill-space/6kdata 2024-09-10T20:07:08.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:08.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj284 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.055 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj285 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj286 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj287 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.229 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj288 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.330 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj289 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj290 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj291 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.572 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj292 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj293 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj294 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.779 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj295 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.839 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj296 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.905 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.906 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj297 td/osd-backfill-space/6kdata 2024-09-10T20:07:09.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:09.972 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj298 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.039 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj299 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.105 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj300 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.172 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj301 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.230 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj302 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj303 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj304 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj305 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj306 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.539 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj307 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.614 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj308 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.681 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj309 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.747 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.748 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj310 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.814 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj311 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj312 td/osd-backfill-space/6kdata 2024-09-10T20:07:10.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:10.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj313 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.014 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj314 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.072 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj315 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.131 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj316 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.197 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.198 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj317 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.256 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj318 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj319 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj320 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.514 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj321 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.573 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj322 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.631 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj323 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj324 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.764 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj325 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.823 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj326 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.881 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj327 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.939 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj328 td/osd-backfill-space/6kdata 2024-09-10T20:07:11.997 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:11.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj329 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.056 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj330 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.114 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj331 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj332 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.231 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj333 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.289 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj334 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.348 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj335 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.406 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj336 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj337 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj338 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj339 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.639 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj340 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj341 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.771 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj342 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj343 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj344 td/osd-backfill-space/6kdata 2024-09-10T20:07:12.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:12.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj345 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.006 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj346 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj347 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj348 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj349 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.238 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj350 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.339 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj351 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj352 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj353 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.588 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj354 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj355 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj356 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj357 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.830 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj358 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.889 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj359 td/osd-backfill-space/6kdata 2024-09-10T20:07:13.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:13.947 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj360 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.005 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj361 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.063 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.064 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj362 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.122 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj363 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.180 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj364 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.239 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj365 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.297 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj366 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.356 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj367 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.414 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj368 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.472 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj369 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.531 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj370 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.589 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj371 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.647 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj372 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.706 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj373 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.806 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj374 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.872 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj375 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.930 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj376 td/osd-backfill-space/6kdata 2024-09-10T20:07:14.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:14.989 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj377 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.048 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj378 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.106 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj379 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.164 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj380 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.223 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj381 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.306 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj382 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.373 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj383 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj384 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.522 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj385 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.581 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj386 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj387 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.698 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj388 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.772 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj389 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.831 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj390 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.890 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj391 td/osd-backfill-space/6kdata 2024-09-10T20:07:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:15.948 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj392 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.007 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj393 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.065 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj394 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.123 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj395 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj396 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.240 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj397 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.298 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj398 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj399 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj400 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.473 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj401 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj402 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.590 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj403 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.648 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj404 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj405 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.765 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj406 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj407 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.882 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj408 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.940 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj409 td/osd-backfill-space/6kdata 2024-09-10T20:07:16.998 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:16.999 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj410 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.057 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj411 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.115 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj412 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.173 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.174 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj413 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.232 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj414 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj415 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj416 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.523 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.524 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj417 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.582 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj418 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.640 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj419 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.699 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj420 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj421 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.840 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj422 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.898 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj423 td/osd-backfill-space/6kdata 2024-09-10T20:07:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:17.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj424 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj425 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.073 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj426 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj427 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.190 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj428 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.248 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj429 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.307 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj430 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj431 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj432 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.482 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj433 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.540 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj434 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.598 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.599 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj435 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.657 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj436 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj437 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.789 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj438 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.849 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj439 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.907 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj440 td/osd-backfill-space/6kdata 2024-09-10T20:07:18.965 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:18.966 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj441 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.024 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj442 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.082 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj443 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.140 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj444 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj445 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.257 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj446 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj447 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj448 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.515 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj449 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj450 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj451 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj452 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.749 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj453 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.807 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj454 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.865 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.866 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj455 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.932 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj456 td/osd-backfill-space/6kdata 2024-09-10T20:07:19.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:19.991 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj457 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.049 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj458 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.107 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj459 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.165 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj460 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj461 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj462 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.340 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj463 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj464 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj465 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj466 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj467 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj468 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj469 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.782 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.783 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj470 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.841 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj471 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.899 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj472 td/osd-backfill-space/6kdata 2024-09-10T20:07:20.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:20.957 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj473 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.015 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj474 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj475 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.132 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj476 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj477 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj478 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj479 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.441 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj480 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj481 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj482 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.632 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj483 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj484 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj485 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj486 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj487 td/osd-backfill-space/6kdata 2024-09-10T20:07:21.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:21.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj488 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.008 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj489 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.066 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj490 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.124 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj491 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.182 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.183 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj492 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.241 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj493 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.299 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj494 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.357 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.358 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj495 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.416 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj496 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.474 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj497 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.532 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.533 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj498 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.591 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.592 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj499 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.649 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj500 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.707 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj501 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.766 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj502 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.824 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj503 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj504 td/osd-backfill-space/6kdata 2024-09-10T20:07:22.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:22.941 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj505 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.021 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj506 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.083 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj507 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.141 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj508 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.199 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj509 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.258 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj510 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.349 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj511 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.432 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj512 td/osd-backfill-space/6kdata 2024-09-10T20:07:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:23.732 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj513 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj514 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj515 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.225 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj516 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.282 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj517 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj518 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.399 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj519 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj520 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj521 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj522 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj523 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj524 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.773 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.774 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj525 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.832 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.833 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj526 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.891 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj527 td/osd-backfill-space/6kdata 2024-09-10T20:07:24.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:24.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj528 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj529 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj530 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.149 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj531 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.208 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj532 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.274 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj533 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.341 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj534 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.449 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.450 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj535 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj536 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.574 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj537 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj538 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj539 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj540 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.825 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj541 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.883 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj542 td/osd-backfill-space/6kdata 2024-09-10T20:07:25.949 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:25.950 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj543 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.016 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.017 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj544 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj545 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.133 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj546 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.191 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.192 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj547 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.249 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.250 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj548 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.308 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj549 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj550 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.425 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj551 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.483 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj552 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.541 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj553 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.600 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj554 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.658 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj555 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj556 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.799 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.800 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj557 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.858 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj558 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.916 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj559 td/osd-backfill-space/6kdata 2024-09-10T20:07:26.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:26.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj560 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj561 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.100 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj562 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.166 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.167 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj563 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.236 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj564 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj565 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj566 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj567 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.583 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.584 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj568 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.650 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj569 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj570 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.791 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj571 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.850 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj572 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.908 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj573 td/osd-backfill-space/6kdata 2024-09-10T20:07:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:27.975 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj574 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.033 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj575 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj576 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.150 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj577 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.209 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj578 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj579 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.333 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj580 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.392 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj581 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.458 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj582 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.516 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.517 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj583 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.575 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj584 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.633 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj585 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.700 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj586 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.758 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj587 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.817 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj588 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.875 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj589 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.934 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj590 td/osd-backfill-space/6kdata 2024-09-10T20:07:28.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:28.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj591 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.050 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj592 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.158 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.159 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj593 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.217 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj594 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.275 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj595 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.350 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj596 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj597 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.508 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj598 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.566 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.567 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj599 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:599: TEST_backfill_grow: for i in $(seq $(expr $rmobjects + 1) $objects) 2024-09-10T20:07:29.625 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:601: TEST_backfill_grow: rados -p test put obj600 td/osd-backfill-space/6kdata 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:604: TEST_backfill_grow: activate_osd td/osd-backfill-space 0 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/osd-backfill-space 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/osd-backfill-space/0 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-10T20:07:29.685 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/osd-backfill-space/0' 2024-09-10T20:07:29.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/osd-backfill-space/0/journal' 2024-09-10T20:07:29.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2024-09-10T20:07:29.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2024-09-10T20:07:29.686 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/osd-backfill-space' 2024-09-10T20:07:29.686 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/osd-backfill-space/$name.log' 2024-09-10T20:07:29.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/osd-backfill-space/$name.pid' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/osd-backfill-space/0 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2024-09-10T20:07:29.689 INFO:tasks.workunit.client.0.smithi130.stdout:start osd.0 2024-09-10T20:07:29.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=bd09a7aa-e48d-40db-9c91-b0cad983a1c8 --auth-supported=none --mon-host=127.0.0.1:7180 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --fake_statfs_for_testing=3686400 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-space/0 --osd-journal=td/osd-backfill-space/0/journal --chdir= --run-dir=td/osd-backfill-space '--admin-socket=/tmp/ceph-asok.115082/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-backfill-space/$name.log' '--pid-file=td/osd-backfill-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-10T20:07:29.690 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/osd-backfill-space/0/whoami 2024-09-10T20:07:29.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2024-09-10T20:07:29.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2024-09-10T20:07:29.690 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2024-09-10T20:07:29.691 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2024-09-10T20:07:29.720 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:07:29.717+0000 7f9074d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:29.721 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:07:29.719+0000 7f9074d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:29.722 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:07:29.720+0000 7f9074d1b640 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-10T20:07:30.074 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2024-09-10T20:07:30.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2024-09-10T20:07:30.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2024-09-10T20:07:30.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2024-09-10T20:07:30.075 INFO:tasks.workunit.client.0.smithi130.stdout:0 2024-09-10T20:07:30.075 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2024-09-10T20:07:30.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:30.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2024-09-10T20:07:30.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:30.076 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:30.502 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:30.815 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:07:30.813+0000 7f9074d1b640 -1 Falling back to public interface 2024-09-10T20:07:31.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:31.504 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:31.504 INFO:tasks.workunit.client.0.smithi130.stdout:1 2024-09-10T20:07:31.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2024-09-10T20:07:31.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:31.505 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:31.910 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2024-09-10T20:07:32.660 INFO:tasks.workunit.client.0.smithi130.stderr:2024-09-10T20:07:32.656+0000 7f9074d1b640 -1 osd.0 23 log_to_monitors true 2024-09-10T20:07:32.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2024-09-10T20:07:32.911 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2024-09-10T20:07:32.912 INFO:tasks.workunit.client.0.smithi130.stdout:2 2024-09-10T20:07:32.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2024-09-10T20:07:32.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2024-09-10T20:07:32.912 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2024-09-10T20:07:33.385 INFO:tasks.workunit.client.0.smithi130.stdout:osd.0 up in weight 1 up_from 29 up_thru 0 down_at 26 last_clean_interval [5,23) [v2:127.0.0.1:6802/2207049855,v1:127.0.0.1:6803/2207049855] [v2:127.0.0.1:6804/2207049855,v1:127.0.0.1:6805/2207049855] exists,up 7c20ffb8-8a7e-4f20-a3dd-350794fdfda1 2024-09-10T20:07:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2024-09-10T20:07:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2024-09-10T20:07:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2024-09-10T20:07:33.385 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:606: TEST_backfill_grow: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-10T20:07:33.600 INFO:tasks.workunit.client.0.smithi130.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-10T20:07:33.608 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:608: TEST_backfill_grow: sleep 2 2024-09-10T20:07:35.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:610: TEST_backfill_grow: wait_for_clean 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: wait_for_clean: local cmd= 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1634: wait_for_clean: local num_active_clean=-1 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1635: wait_for_clean: local cur_active_clean 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: get_timeout_delays 90 .1 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: shopt -q -o xtrace 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: echo true 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: get_timeout_delays: local trace=true 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: true 2024-09-10T20:07:35.612 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: get_timeout_delays: shopt -u -o xtrace 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1636: wait_for_clean: local -a delays 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1637: wait_for_clean: local -i loop=0 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1639: wait_for_clean: flush_pg_stats 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2205: flush_pg_stats: local timeout=300 2024-09-10T20:07:35.708 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ceph osd ls 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2207: flush_pg_stats: ids='0 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr:1 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr:2' 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2208: flush_pg_stats: seqs= 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:07:36.103 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-10T20:07:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=124554051587 2024-09-10T20:07:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 124554051587 2024-09-10T20:07:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587' 2024-09-10T20:07:36.224 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:07:36.224 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-10T20:07:36.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=42949673008 2024-09-10T20:07:36.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 42949673008 2024-09-10T20:07:36.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-42949673008' 2024-09-10T20:07:36.344 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: flush_pg_stats: for osd in $ids 2024-09-10T20:07:36.344 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-10T20:07:36.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2210: flush_pg_stats: seq=64424509480 2024-09-10T20:07:36.464 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: flush_pg_stats: test -z 64424509480 2024-09-10T20:07:36.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: seqs=' 0-124554051587 1-42949673008 2-64424509480' 2024-09-10T20:07:36.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:07:36.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 0-124554051587 2024-09-10T20:07:36.465 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:07:36.465 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=0 2024-09-10T20:07:36.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 0-124554051587 2024-09-10T20:07:36.466 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:07:36.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=124554051587 2024-09-10T20:07:36.467 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.0 seq 124554051587 2024-09-10T20:07:36.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-09-10T20:07:36.467 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:07:36.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 1 -lt 124554051587 2024-09-10T20:07:36.862 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: sleep 1 2024-09-10T20:07:37.863 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2224: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-10T20:07:37.863 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-10T20:07:38.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 124554051587 -lt 124554051587 2024-09-10T20:07:38.262 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:07:38.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 1-42949673008 2024-09-10T20:07:38.262 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:07:38.263 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=1 2024-09-10T20:07:38.263 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 1-42949673008 2024-09-10T20:07:38.264 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:07:38.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=42949673008 2024-09-10T20:07:38.265 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-10T20:07:38.265 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.1 seq 42949673008 2024-09-10T20:07:38.265 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-10T20:07:38.670 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 42949673009 -lt 42949673008 2024-09-10T20:07:38.671 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: for s in $seqs 2024-09-10T20:07:38.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: echo 2-64424509480 2024-09-10T20:07:38.671 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: cut -d - -f 1 2024-09-10T20:07:38.672 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: osd=2 2024-09-10T20:07:38.672 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: echo 2-64424509480 2024-09-10T20:07:38.673 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: cut -d - -f 2 2024-09-10T20:07:38.673 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: flush_pg_stats: seq=64424509480 2024-09-10T20:07:38.674 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: echo 'waiting osd.2 seq 64424509480' 2024-09-10T20:07:38.674 INFO:tasks.workunit.client.0.smithi130.stdout:waiting osd.2 seq 64424509480 2024-09-10T20:07:38.674 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-10T20:07:39.185 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: flush_pg_stats: test 64424509481 -lt 64424509480 2024-09-10T20:07:39.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: get_num_pgs 2024-09-10T20:07:39.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:39.186 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:39.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1640: wait_for_clean: test 1 == 0 2024-09-10T20:07:39.856 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:39.856 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:39.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:39.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:39.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:39.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:39.857 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:40.331 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:40.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:40.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:40.331 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:40.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:40.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' -1 2024-09-10T20:07:40.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: loop=0 2024-09-10T20:07:40.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1652: wait_for_clean: num_active_clean=0 2024-09-10T20:07:40.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:40.903 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:41.004 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:41.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:41.005 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:41.396 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:41.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:41.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:41.396 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:41.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:41.992 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:41.993 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:42.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=26111 2024-09-10T20:07:42.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 26111 '!=' null 2024-09-10T20:07:42.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:42.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:42.601 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:42.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:42.702 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:42.703 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:43.168 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:43.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:43.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:43.169 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:43.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:43.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:43.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:43.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:43.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:43.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:43.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:43.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:43.714 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=61452 2024-09-10T20:07:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 61452 '!=' null 2024-09-10T20:07:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:44.264 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:44.366 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:44.754 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:44.754 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:44.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:44.755 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:45.342 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:45.343 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:45.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=52716 2024-09-10T20:07:45.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 52716 '!=' null 2024-09-10T20:07:45.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:45.901 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:45.902 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:46.003 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:46.391 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:46.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:46.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:46.391 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:47.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:47.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:47.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:47.012 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:47.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:47.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:47.013 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:47.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:47.013 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=113150 2024-09-10T20:07:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 113150 '!=' null 2024-09-10T20:07:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:47.611 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:47.712 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:47.713 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:48.145 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:48.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:48.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:48.145 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:48.704 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:48.705 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:49.245 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=113150 2024-09-10T20:07:49.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 113150 '!=' null 2024-09-10T20:07:49.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:49.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:49.246 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:49.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:49.347 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:49.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:49.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:49.347 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:49.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:49.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:49.348 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:49.798 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:49.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:49.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:49.798 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:50.371 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:50.372 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=95744 2024-09-10T20:07:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 95744 '!=' null 2024-09-10T20:07:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:50.959 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:51.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:51.060 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:51.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:51.060 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:51.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:51.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:51.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:51.061 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:51.447 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:51.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:51.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:51.448 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:52.025 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:52.026 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:52.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=95744 2024-09-10T20:07:52.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 95744 '!=' null 2024-09-10T20:07:52.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:52.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:52.615 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:52.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:52.716 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:52.716 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:52.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:52.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:52.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:52.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:52.717 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:53.178 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:53.178 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:53.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:53.179 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:53.714 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:53.715 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:54.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=168962 2024-09-10T20:07:54.321 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 168962 '!=' null 2024-09-10T20:07:54.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:54.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:54.322 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:54.423 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:54.808 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:54.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:54.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:54.808 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:55.456 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:55.457 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:55.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:55.457 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:55.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=135682 2024-09-10T20:07:55.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 135682 '!=' null 2024-09-10T20:07:55.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:55.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:55.990 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:56.091 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:56.092 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:56.467 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=0 2024-09-10T20:07:56.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:56.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:56.468 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 = 1 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: test 0 '!=' 0 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: get_is_making_recovery_progress 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2024-09-10T20:07:56.995 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-10T20:07:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=175872 2024-09-10T20:07:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test 175872 '!=' null 2024-09-10T20:07:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: loop=0 2024-09-10T20:07:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: eval 2024-09-10T20:07:57.587 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: sleep 0.1 2024-09-10T20:07:57.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: loop+=1 2024-09-10T20:07:57.688 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: true 2024-09-10T20:07:57.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: get_num_active_clean 2024-09-10T20:07:57.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2024-09-10T20:07:57.688 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-10T20:07:57.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-10T20:07:57.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2024-09-10T20:07:57.689 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-10T20:07:58.070 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: cur_active_clean=1 2024-09-10T20:07:58.070 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-10T20:07:58.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2024-09-10T20:07:58.071 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2024-09-10T20:07:58.605 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 = 1 2024-09-10T20:07:58.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: break 2024-09-10T20:07:58.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: return 0 2024-09-10T20:07:58.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:612: TEST_backfill_grow: delete_pool test 2024-09-10T20:07:58.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: delete_pool: local poolname=test 2024-09-10T20:07:58.606 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:538: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-10T20:07:59.081 INFO:tasks.workunit.client.0.smithi130.stderr:pool 'test' does not exist 2024-09-10T20:07:59.092 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:613: TEST_backfill_grow: kill_daemons td/osd-backfill-space 2024-09-10T20:07:59.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:07:59.093 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:07:59.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:07:59.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:07:59.093 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:08:04.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:08:04.424 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:614: TEST_backfill_grow: grep -q 'num_bytes mismatch' td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:614: TEST_backfill_grow: return 1 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh:38: run: return 1 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: code=1 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: teardown td/osd-backfill-space 1 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/osd-backfill-space 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2024-09-10T20:08:04.430 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/osd-backfill-space KILL 2024-09-10T20:08:04.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2024-09-10T20:08:04.431 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2024-09-10T20:08:04.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2024-09-10T20:08:04.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2024-09-10T20:08:04.431 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2024-09-10T20:08:04.436 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2024-09-10T20:08:04.436 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2024-09-10T20:08:04.437 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-10T20:08:04.437 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2024-09-10T20:08:04.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-10T20:08:04.439 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2024-09-10T20:08:04.439 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2024-09-10T20:08:04.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:08:04.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2024-09-10T20:08:04.440 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2024-09-10T20:08:04.441 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:08:04.442 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:08:04.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2024-09-10T20:08:04.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2024-09-10T20:08:04.443 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2024-09-10T20:08:04.444 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/osd-backfill-space/mgr.x.log td/osd-backfill-space/mon.a.log td/osd-backfill-space/osd.0.log td/osd-backfill-space/osd.1.log td/osd-backfill-space/osd.2.log /home/ubuntu/cephtest/archive/log 2024-09-10T20:08:04.445 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/osd-backfill-space 2024-09-10T20:08:04.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2024-09-10T20:08:04.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2024-09-10T20:08:04.459 INFO:tasks.workunit.client.0.smithi130.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.115082 2024-09-10T20:08:04.459 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.115082 2024-09-10T20:08:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:08:04.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2024-09-10T20:08:04.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2024-09-10T20:08:04.461 INFO:tasks.workunit.client.0.smithi130.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: main: return 1 2024-09-10T20:08:04.462 INFO:tasks.workunit:Stopping ['osd-backfill'] on client.0... 2024-09-10T20:08:04.462 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T20:08:04.755 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh' 2024-09-10T20:08:04.979 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f72ca32c84cf4ab68280af42990c0ffb Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi130 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh' 2024-09-10T20:08:04.981 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T20:08:04.990 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T20:08:04.990 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:08:05.043 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T20:08:05.061 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T20:08:05.061 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:08:05.076 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-09-10T20:08:05.080 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T20:08:05.080 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-09-10T20:08:05.081 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv01.front.sepia.ceph.com 3 6 377 11 -823us[ -820us] +/- 46ms 2024-09-10T20:08:05.081 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 5 +2246us[+2246us] +/- 58ms 2024-09-10T20:08:05.081 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 -1885us[-1881us] +/- 80ms 2024-09-10T20:08:05.081 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T20:08:05.082 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T20:08:05.090 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T20:08:05.091 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T20:08:05.098 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T20:08:05.180 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T20:08:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:08:05.207 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T20:08:05.215 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T20:08:05.224 INFO:teuthology.task.internal:Duration was 2241.948968 seconds 2024-09-10T20:08:05.224 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T20:08:05.232 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T20:08:05.232 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:08:05.288 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T20:08:05.775 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T20:08:05.776 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-09-10T20:08:05.776 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:08:05.839 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T20:08:05.839 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:08:05.907 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T20:08:05.907 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:08:06.044 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T20:08:06.052 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T20:08:06.053 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:08:06.077 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T20:08:06.088 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:08:06.140 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-09-10T20:08:06.164 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:08:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:08:06.219 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T20:08:06.228 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T20:08:06.229 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899489/remote/smithi130 2024-09-10T20:08:06.230 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:08:16.297 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T20:08:16.298 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:08:16.379 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T20:08:16.388 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T20:08:16.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T20:08:16.397 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T20:08:16.397 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:08:16.414 INFO:teuthology.orchestra.run.smithi130.stdout: 262169 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 20:08 /home/ubuntu/cephtest 2024-09-10T20:08:16.414 INFO:teuthology.orchestra.run.smithi130.stdout: 405775 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 19:36 /home/ubuntu/cephtest/mnt.0 2024-09-10T20:08:16.414 INFO:teuthology.orchestra.run.smithi130.stdout: 405776 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 20:03 /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T20:08:16.414 INFO:teuthology.orchestra.run.smithi130.stdout: 405819 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Sep 10 20:03 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:08:16.414 INFO:teuthology.orchestra.run.smithi130.stdout: 420643 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:08 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2024-09-10T20:08:16.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:08:16.415 INFO:teuthology.orchestra.run.smithi130.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-09-10T20:08:16.416 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T20:08:16.424 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T20:08:16.433 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T20:08:16.442 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi130 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-09-10T20:08:16.446 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd-backfill} duration: 2241.9489681720734 failure_reason: 'Command failed (workunit test osd-backfill/osd-backfill-space.sh) on smithi130 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd-backfill/osd-backfill-space.sh''' flavor: default owner: scheduled_skanta@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f72ca32c84cf4ab68280af42990c0ffb status: fail success: false 2024-09-10T20:08:16.446 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:08:16.532 INFO:teuthology.run:FAIL